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I» INTRODUCTION 



These little soldiers marching out 
Could put the bravest foe to rout 
And place the world beneath their sways 
But since their arms are rather light. 
If I interpret it aright, ^ 
They ■ re only setting out to play , 

CIASSES OF MODELS, glHmATIONS, AND GAMES 

Complex problems often demand complex analytic techniques.' gaming 
and simulation are two such methods. .Generally characterized as having 
not only intuitive appeals-abased on nice descriptive properties^^but 
also a solid e-mpirical basis , these methods have stimulated the devel- 
opment of a large and expanding professlonMl community and literature. 
What this community does, the trends and size of its activity ^ and where 
it seeme to be headed are questions of importance. 

Even after many years of works it is still difficult to state with 
precision what is meant by gaming and simulation* What start out as 
games J for example, may degenerate into fruitless syntactic exercises. 
However, it is useful to distinguish four categories: analytic models, 
machine simulations, man-tnachine games, and free-form games. 

Analy tic Models 

A good analytic model is usually quite abstract , poor in the num-- 
ber of variables explicitly considered but rich in ease of manipulation 
and clarity of insight. For many questionSj the analytic model may 
give a single number for an answer, as contrasted with multiple, inter- 
related indicators of system behavior that may result from the use oi 
other techniques* Truej more than a single number, a kill probability 
or a specific survival level, for instance, may be of analytic interest. 
Frequently, howevers a single end^state condition for a system is 

Stanley Appelbaum, trans,, "War," Gmm$ mid Pastimes of Child- 
hood^ New York- Dover, 1969, pi, 26; originally published as Jacques 
^ ' Stella, L08 Jeucc et Plaisire de l-Enfanae^ Paris, 1657. 
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calculated. This contrasts with the study of system bahavior Chrough 
time, in which end Etates may not be of interest or even relavant. 

Certain fomis of warfare have been characterized by game^theoretiCs 
analytic models in which two-^sided or more than two-^sided combat is 
considered explicitly. For example ^ one might wish to examine the op- 
timal behavior of a red team versus a blue team when neither sldg-s 
strategy is fixed. In a situation such as a two-^sided war, one would 
think that assuming pure opposition would enable one to consider opti- 
mal tactics s and that deductions of an opposition's best strategy could 
lead to the formulation of optimal strategies. Unfortunately j it is 
generally not possible to do so* Combat situations more complex than 
simple tactical encounters are frequently not well represented as "Eero^ 
.sum games" because they may not be situations of pure opposition, Fur^ 
thermorei elementi omitted from the analysis in the interest of tract^ 
ability and precision (for example j "human factors*') may be crucial to 
understanding what is in fact going on. 

Good analytic models help spot the "chicken and egg'' problem thatj 
once recognised s can usually be solved. The point bears directly on 
the relationship among rigorous theoretical models, empiricisms and 
data gathering. An analytic model is usually too restricted to solve 
an actual operational problem directly. But because a model is norm= 
ally clean and clear, it can warn about potential difficulties, Indicate 
where additional measurements are most needed, and identify and order 
Important omiasionii This preaupposes, of course, that those preparing 
the analytic modeli communicate with those who have the oparatlonal 

problem, and that their findings are received and understoodi 

_ t _ 

The op timal-^asslgnment method created by John von Neumann is an 
excellent example of an analytic model that has limited interest for 

<^ 

Fur example, Melvin Dresher, Gmes of Stpategy' Thmry and Ap^ 
pliaatu^nB, Englewood Cliffs, J.: Prentice-Hall, 1901, 

John von Neumann, "A Certain Two-Person, Zero-sum Game Equivalent 
Co the Optimal Assignment Hroblems" in H, W, Kuhn and A, W. Tucker ^ 
eds,, Annala of Maihematiaal Study ^ No, 28, Princeton University Press, - 
1953, pp. 12-15. 



-3- 



dlrect operational application but that provides many basic 'insights * 
ABSuming that one knows yxactly what a man can and will dos that model 
considers a set of tasks that must be done and offers a ^way^to~ assign 
men to tasks in an optimal manner. Though the simplifying assumptions 
may rtduce the model ' s direct operational application and utility, 
von Neumann's work is clean and abstractly captures the core of an im- 
portant problem. 

Analytic work on the Berlin airlift provides an example where ini- 
tially simple analysis yielded to more complex forinulations as the prob- 
1cm began to be understood. From the first linear programj the anal^ 
ysis evolved to dynamic models and ended up as a combination dynamic 
model nnd simulation * The evolutionary process was probably , in a 
strict pragmatic senses optimal. One might conjecture that until the 
problem was "learned*' with simpler, more abstract analyses, realistic 
representation was just not possible^ 

People most at home with analytic models Include applied mathema^ 
ticiansi operationa research analysts, and a breed that may be called 
'■computeruiks, although the computer is frequently used merely as an 
analytic aid. 

Growth in the use and power of analytic models over the last 30 
years has been astounding. In techniques, new insights, and amount 
of personnel, that growth has probably exceeded all previous work. The 
prospects are good for more diverse applications of analytic models and 
for increased use of the computer, not as a simulator but as an aid in 
solving analytic problems* The outlook is bright for serious model'- 
building, problem-solvlngi ' and careful application* The status of and 
prospeets for the three other categories are less certsini 

Machine Smulations 

In contrast with analytic models , machine simulations frequently 
involve many variables; many seem to make a fetish of "realism*'' Ra= 
tionales for doing simulations are many and varied, One common and 

A. S, Manne, Allooating MATS Equipment with the Aid of Lim<w 
Pj'ogrmmning ^ The Rand Corporatloni IU^-1612, January 1956 . 



frequentry valid reason is that mathematics is a relatively Impoverished 
language, whereas the computer allows one to capture the richness or 
robustness of a real system. What is left unstated is the enormous 
price one is likely to pay to approKlmate that reality. 

There are literally hundreds of machine simulations in the Depart- 
ment of Defense's current , active inventory. Most of them are force- 
structure, weapon-system, and technical-evaluation models. 

Simulations and the people ("simulators") responsible for their 
design^ Implementation, and operation are much harder to control than 
analytic models or modelers, Not only are fewer scientific gtandards 
available to aid In evaluating a computer simulation than a mathemati- 
cal model, there Is little or no consensus among prdfesslonals as to 
what the proper standards are. 

Promotion may well be the undoing of this potentially highly use- 
ful aipect of the profession. Large-scale computer simulations have 
bean rather easy to sell. They appear reasonable, and they provide a 
wealth of material for impressive, high-level briefings. Advocacy or 
defense of a given system or poinn of view ^ can be made to appear quite 
scientific* 

Machine simulation has been oversold in the last ten years or so, 
and the shakedown is now taking place. The process is healthy j albeit 
painful in certain quarters. Much has been learned that contributes 
to the prof essionaligat ion of work. Problems have been delineated » and 
lines of communication are being established. Still, pTrnctitloners 
must show more care in (1) modeling and ipecif ication, (2) data collec= 
tion and validation, (3) sensitivity analysis and question formulation ^ 
and (4) accounting for costs and utilization. Each of these tasks is 
a challenge by Itself. Together tney make the difference between a 
viablei useful simulation and something that is only useful to generate 
visual aids for high-level "dog and pony show" briefings , 

Man->techlne Games 

Man-machine exercises usually Involve a digital computer and peo- 
ple playing some of the roles in the modeled syitem. People may be 
used merely because they are cheaper than the software. Or, people may 



be used because human factors (particularly judgment) are Important in 
the situation being analyzed. 

In Tnan-machine gaming and simulation, the personnel are usually 
analysts i systems engineers, experimental psychologlsti , social piy^ 
chologistSj and economists. In work emphasizing himian factors, hiflnans 
are studied^ not merely used as inputs. With a systems^engineering em- 
phaals, humans may be used because they are handy and relatively cheap. 
In that case p the orientation is more toward operations research analy-^ 
sis and systems engineering than it is to experimental or iocial psy- 
chology. 

The former Logistics Simulation Laboratory at The Rand Corporation 
is an exmple of a man-machine simulation in which people were used 

more as an integral part of the machinery than as subjects for human- 

* _ _ 

factor analyses, John L, Kennedy -s early work at Rand, on the other 

t 

handp is an example of human-f actore analysis. 

Man-machine gaming and simulation waSp at its peak of activity, 
five to seven years ago, probably oversold* Currently it is undersold, 
if not totally neglected. What is needed is a basic inventory of in- 
forination that stresses substantive content, The connection between 
machine-only simulations, human-f actors analyses, and analytic models 
is slight, to their mutual detriment # 

Free-Form Garolng 

Free-form gaming involves teams and a referee group operating 
within the framework of a scenario* If computational equipment is used 
at all, it is often relegated to a simple bookkeeping role. Of the 
four classes, this one is the least ^enable to tight technical con- 
trol* It is J however J the most likely to produce an Impressive array 
of new insights into complex problems, 

M* A. Geisler and A. S, Ginsberg, Man^MaaMne Smulation Emperi^ 
BnoB^ The Rand Corporation, P-3214, August 1965, 

t 

R» L, Chapman and J, L, Kennedy, The Background and Jmpliaatiom 
of the Systm H&se^ah Laboratory Studi^&^ The Rand Corporation, P-740, 
1955, See also Ni Frederlckson, "Factors in In-Baiket Performance," 
PByQhologiaal MonogvaphB^ No. 22, 1962, for another good example of the 
genre. 
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Free-form gaming Is alao the least axpensive. In fact, campared 
with the expenses associated with machine simulations and models, the 
costs of all other types seem minuscule. A familiar example of free-- 
form gaming will suggest why and the way in which it is least cascly. 

The Political^Military Exercises at and for the Joint Chiefs of 
Staff's Studies, Analysis, and Gaming Agency (SAGA) are aimed at the 
strategic level and Involve relatively high-priced personnel/ Strictly 
speaking, the cost of a game or simulation must Include not only the 
price of the physical resources but also the value oF the time of the 
personnel playing the game. In fact, the costs depend on how one eval- 
uates the worth of the time of the fop-^echelon teams* But if personnel 
(and opportunity) costs are ignoredi as is usual, all that a top-level 
political-military exercise needs are a couple of rooms, a few people 
(who are already in the Pentagon), some pencils, pads of paper, and a 
bit of videotape. Participants in the SAGA games may be Cabinet offi-^ 
cialSj three- or four-star generals, or admirals, yet the amount of 
money involved for the other resources is seldom more than $10,000 per 
game , 

Though free'=form gaming is the least tightly controlled and the 
least expansive type, it receives far and away the most publicity and 
is done at the highest policy level of all four types* Free-form gam- 
ing also has a few good practitioners and a product that is vary hard 
to measure, making it extremely difficult to ascertain whether the art 
form has improved in the last few years* A little more money and some 
careful, rigorous work may yield some useful and productive results, 

NONMILITARY APPLICATIONS OF GAMING AND SlhaJLATIQN 

Before dlscuislng our survey's findings about the military uses of 
models j simulations, and games, some observations on their general uses 
are in order, These remarks are limited to simulations or models tliat 
relate to competitive or cooperative behavior, that Is i the behavior 
nf more than one decisionmaker. Accordingly, studies such as the 



For a clear introduction to this type of game, see Herbert 
Goldhamer and Hans Speier, "Some Observations on Political Gaming," 
Wovld PoUtiao, Vol. 12, 1959, pp, 71-83. 
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simulation of an oil refinery or transportation system are ruled out. 
Relevant applications fall into four main classes: industrials-opera-- 
tlonal, teaching and training, polltical--diplomatlc^militarys and re- 
search, 

Indus trial^^eratlonal 

We have found little use of gaming and simulation involving more ' 
than one decisionmaker in any civilian setting^-'in contrast to what arti- 
cles in BusineBs Weekj Fortune^ and Hay'vavd Business Hevi&u) suggest* 
While business gamei are frequently used In training programs, as enter- 
tainment for executives^ and as teaching devices at business schools, 
the production and use of operational games by industry appear to be 
minimal. Gaining for teaching purposes is particularly active: there 
exist about 500 business games ^ whcse costs vary widely from a few thou^ 
sand to hundreds of thousands of dollars* However j because of inade- 
quate accounting, one cannot estimate the total investment or even the 
current operating costs for such activities,^ 

T eaching and Training 

Recently there has been a spurt in the use of games for introduc- 
tory teaching. At the level of university and adult education, the 
predominant type of game is the business game and its variants, which 
are usually computerized games* One class of such games relates to 
planning and development, including the land-^use planning games CLUG, 
CITY, and >ffiTRO, and the air pollution exercise APEX, Intended to 
teach the management of urban development problems, these games have 
cost hundreds of thousands of dollars in total. The proliferation of 
games concerned with social problams such as congestion, pollution ^ and 
crime seems inevitable. Current funding for these activities is rela- 
tively modest, but the number and diversity of research proposals are 
growing. 

Such activities doubtless exist as strictly proprietary ventures. 
The full scope and magnitude are, because of secrecy, probably unknow- 
able. 

t 

See Martin Shubik, "Gamingj Coits and Facilities,'^ ManagBment 
Saienae, Vol. 14, No. 11, 1968, pp, 629-660, 
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l ^jUcical- n iplumattc-Milltary 

Anotlier type of gaming activity at the univ^tdity or adult-educa- 
tion leval Is political^diplomatic^milicary ganiing. It is Cypically 
of the free-form variety and relatively intaKpensive . An upper bound 
on expend) turtss during thci last 20 years is certainly no mora than 
$2 million per annum, and probably much less. That cruda estlmatti is 
guessed at from the budget of the pDlitical-mill tary branch of SAGA, 
plus the budget of Harold Guetzkow's work on the. InCer-Nation Simula- 
tion, plus a rough sum to account for various smaller activities sucli 
as the work by Lincoln Bloqmfield at M*I*T., and political--diplomatic- 
niilitary gaming at the war colleges, military academies, Rand, the Re- 
sCi.rch Analysis Corporation, and a few universities. The amaunt of 
publicity given free-form, poll t ical-dip lomat Ic-^mili tary games has been 
linormously disproportionaLe to the financial and incellectual Invest- 
ments in them. - Popular accounts, such as Andrew Wilson's recent Th,3 

^vij a^ui Lpmpuu^j:p aside, research on the intellectual foundations 

and uses of this type of work b.as been negiigible. 

Research 

Gaming and competitive simulations devoted to research are concen- 
trated in the universities, are associated with work in social psychol- 
ugy and to some extent with the formal theory of games, and are slowly 
increasing* Depending upon how the research is classified and costedj 
$2-5 million a year is probably being spent on these activities. Ex- 
amples are the work of Anatol Rapoport and his associates, the work of 
Gerald Shure and Harold Kelley at UCLA, and the activltiea of the gam= 
ing laboratories at Berkeley, Purdue, Yale, Ohio State, and other uni-^ 
versities. Specialized research agencies have used gaining as an adjunct 
to research* An Important result of many of these games is the iden-- 
tification of key problems that are subsequentry reaearched by other 
methods. A small amount of basic human-factors analysis is being per- 
formed by a few groups such as the Army's Behavioral Sciences Research 
Laboratories, 



Andrew Wilson^ TJw Bomb and bhs Computer* Wargmiing from Anai&nt 
Ch%n&&e Mapboard to Atomia Computev^ New York,* Delacorte PresSi 1968, 



Onci could assign a certain percentage of tha budgeEs for accivicieH 
in the foregoing classes to researclu However, it is practically ini- 
pnRslbie to figure the appropriate percentage, and there are strnng in- 
dications that/'such research is not being undertaken. 

•i ' 

Gjme Theory 

One further topic is the role of mathematical, game^ theory model 
construction in the study of conflict. In the United States a handful 
of professionals have specialiEed in the investigation of dueJs and al-- 
location and search games. This work is a mathematical art form that 
has added to the basic knowledge about competitive situations at a rela^ 
tively low cost* These few prof assionals require no extraordinary equip-- 
ment; hence ^ keeping them fully employed amounts to providing salaries 
and standard overheads, A liberal upper^bound estimate on total expendi- 
tures for such game-theoretic analyses of conflict situations is ?l-2 
million a year. 

Summary 

The major activities and eKpenditures in operational gaming and' 
simulation to study conflict and cooperation are by the military. Such 
expenditures elsewhere are negligible. Expenditures for teaching- 
training gaming for university and adult education amount to no more 
than a few million dollars but are undoubtedly growing. Gaming in in- 
troductory education appears to be expanding as well. In research, ex-^ 
perimental activity at the universities is increasing^ but excapt for 
a few corporate-sponsored projects at places such as Rand and RAC, there 
appears to be very little direct funding of basic research elsewhere. 
That includes the military, except for a small amount of human-factors 
analysis . 

Publicity devoted to gaming and simulation appears to vary inversely 
with actual expenditures and activities* High-level political-military 
eKercises may have- caught the imagination of the public, but tliey nave 

M. Dresher, J. Mayberry, S, Moglewer, L, Shapley , and several 
others coma to mind* 



not caught even a minute fraction of the total resources being devoted 
to gaming and simulation. 

Professional Interest and activity in gaming and competitive sim- 
ulation is reflected in the membership of the Military Operations Re^ 
search Society (MORS), which has some 4400 members on its current mail^ 
ing list (down from a peak of about 5500 several years ago) , and of the 
Institute of Management Science's College on Gaming and Simulation, 
which had a 1971 membership of around 650 (slightly larger than several 
years ago). 

The growing literature is another indicator of activity. Of course 
it shows a pronounced bias toward the nonmilitary aspects of gaming and 
simulation because military studies, even if they are not classified, 
normally do not appear in the published literature. Of the approxi- 
mately 2000 articles and books that were Investigated in the course of 
the authors' recent research , about 15 percent were operational and less 
than 12 percent were devoted to war gaming and political-diplomatic^ 
military gaming. 

Historically it appears that activity and expendi tares on gaming 
and competitiva simulation hit a peak in the early to middle 1960s and 
have bee/' on a gentle decline since then, 

A survey and evaluation of the lltei^ature are contalnad in Martin 
Shublkj Garry D, Brewer ^ and Eve Savage, Thm LiteratuTB of Gcm^ng^ Sim^ 
ulation^ and Model^Duilding: Ind&m and Critiaat AbBtraate^ The Rand 
Corporation, R-620-ARPA, 1972, and Martin Shubik and Garry D, Brewer, 
Reviews of Seteoted Books and Ar^tial^a on Gmning and Simulation^ The 
Rand Corporation, R-732-AWA, 1972. 
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II, THE SURVEY AND ITS RESULTS 



Having ascertained that the major expanditures are devoted to com^ 
petltive, all^-machine simulations and models , considerably less to man- 
machine games, and the least amounts to free-form games, we identified 
approKimately 450 active military models | simulations, and games of 
those types in several Department of Defense and service catalogs and 
inventories p In close collaboration with the General Accounting Office, 
which had i^ieppndently been asked by the House Appropriations Committee 
to assess the iind, extent, and use of military gaming, we initially 
chose a sample of ISO models, simulations * and games, which number was 
later pared down to iJ5. A detailed, 70--page questionnaire* was circu- 
lated to the 135 groupLVj agencies, or individuals in DOD that were 
judged to be most knowledgeable about each model, simulation, or game." 
We received 133 replies, one of which gave so little Information that 
it was unusable. Thus, unless otherwlie notedj the itatistical ma* 
terlal below Is based on a respondent population of 132. Replies were 
coded for computerization (see the codebook. Appendix C) , with multiple 
checks on the accuracy of transcription. 

The a^ple was chosen to include several of the larger or more 
heavily used simulations and games. The ample was also biased somewhat 
tqwardi the Army, thare being 59 Amy, 26 Air Forces 35 Navy, and 11 
other DOD personnel in the population. Extrapolations from this samplo 
to the total 450 models , simulations, and games should be made carefully, 
keeping these blasee In mind. 

Three preliminary varslone of the questionnaire were developed smd 
- tried out on known games, utilising cooperative and competent respondents. 
Our thanks to E. W. Paxson, J, R» Llnap H, Guetikow, M. G. Walner, and A» 
Theony for their constructive assistance with this critical portion o£ 
the research. The final version of the queetlonnaire was published ai 
Martin Shublk and Garry D, Brewer, QueBtioTmcdps : ModmlB^ Computer Ma^ 
ohine SwiutationQ^ Gmee and Studies^ The Rand Corporation, F-4672, July 
1971. It is attached as Appendix A, filled out with sifflmary results of 
the survey, and seives as an organizing device for the body of this re- 
port, 

t 

Appendix B provides a full list of the names of the models, simu- 
lations, and g^es surveyed in the preliminary and final stages of this 
project. 



Total reported development costs war© approximately 9 32 milltun, 
but approximately 30 responses did not provide appropriace or clear 
cost informatioru Tney included several of what are believed to he 
the larger simulations. Thus, by siinple averaging, total costs approx- 
imate $40 million. Since the sample represents around 30 percent of 
the total inventory^ a crude estimate of total investment is $130-$140 
million for all active models, simulations, and games in the DOD inven- 
tory. Using other costing criteria before our questionnaire was circu- 
lated, the General Accounting Office made a preliminary estimate of 
$170,5 million, which seems a plausible upper bound. On the average, 
the inventory represents a thrae-^ to four-year supply (an average model, 
simulation, or game "lasts'* three to four years); hence we estimate that 
$30 to $40 million per year has been expended recently for construction 
costs. These estimates are of necessity very rough. In fact, one of 
our major findings concerns the poor to nonexistent cost-accounting 
definitions and procedures. 

The size of expenditures and number of activities do not provide 
all the clues to the importance of the work. Many other criteria are 
needed to judge these activities. But valid criteria have not yet been 
formulated for the profession. With these caveats in tnlnd, let us turn 
to a broad description of the survey results, 

THE RESPONDENTS 

When the final version of the questionnaire was set, the General 
Accounting Office met with personnel at DOD "points of contact,'' de- 
scribed the rationale of the survey, and asked that the most knowledga- 
ftble person about any given model, simulation, or gme be assigned to 
answer each questionnaire. If one person could not do it, groups or 
committaes were acceptable substitutes as long as the responses were 
the most up-to-date and accurate possible. 

As It turned out, most questionnaires were filled out by one person, 
and most respondents filled out only one questionnaire* In a few 



ERIC 



U,S, General Accounting Office, Computer Smiilationa^ Wa^ Gming 
md Contraat Studies^ A Report to the Committee on Appropriations, 
O , House of Represantatlves, Washington, D,C., February 23, 1971, p/s* 
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instances, one man was responsible for two or perhaps three separate 
entries. One respondent did yeoman service on six or seven question- 
naires. 

Respondents were mostly users, 42 or 31.8 percent, or designer/ 
builders, 41 or 31»1 percent. The next largest classes of respondents 
were funder/sponBors , 19 or 14 , percent, and Caretakers , 13 = or 13,6 
percent * 

The average length of time required to fill out the questionnaire 
was 10*5 hours* The mocial time was 4 hours j and 84 were able to answer 
in 8 hours or less; however, two or three respondents stated that they 
took as much as 70 hours * 

One might thus presume that this group of respondents would be 
able to provide, as well as or better than any other group that might 
be formed, reasoned and valuable replies to the questionnaire* The 
following pages discuss their replies, grouped under the headings pur-- 
posess production, operations, use, and costs of modelss simulations^ 
and games, and their opinions about certain hypothetical Innovations 
in the gaming profession , 

PURPOSES 

Because there is so little agreement even about fundamental defi- 
nitions ^ it seemed especially Important to find out what terms and con- 
cepts the professional community is currently using to describe the 
rationale or intent of various models, simulations, and games The 
intention or purpose for which an MSG is built has a direct and im- 
portant bearing on how one subsequently goes about specifying, vall^ 
datingi and controlling it. 

The name, affiliation, and address "of each respondent have been 
coded and Included in the data; however, that information ii not re-- 
ported here, 

"Reported time to complete each questionnaire is listed in Appen- 
dix 

.For convenience, the authors have expressed the notion of models 
simulation, or game as "MSG," which term is used hereinafter. 
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Basle Categories 

We began by asking each reapondenC to identify his MSG as either 
a nodel, simulation, a man-nachine or manual game, a mathematical anal- 
ysis or study, or, if none of those categories fit, as an "other," Re- 
sults are shown in Pig. 1. Moat were identified as models. The usage 
of that term and concept is vague, however; one respondent noted that 
to him a model meant any regression equation, therefore the universe 
of models Is so large as to be virtually unbounded. In short, the 
word has little referential utility. The terminological problem is un- 
derscored by the General Accounting Office's initial lack of success 
when it asked the military services to provide data on "war games." The 
GAO found that practically no war games are played; rather, models and 
simulations are built. 



^^^^^^ m^ f Mm^mmmmfym^ (46) 34.8% 



Model " ^ ^ _ . _ - 



Simulation 



ME^H^ (11) 8.3% 



Man-machine game 
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Manual game 

i (4) 3.0% 
Analyils 
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Other 
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Fig, 1 —Respondents' primary cIcBsificatlon of their MSGs 
(variable CATEG 1 )* 



*See Appendixes A and C for complete definitions of all variables. 
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Another striking finding is how few nmn^machine and manual ex^r^ 
cises are in the DOD's active inventory (14 or 10.6 percent)* Wliile 
our sample represents 132 of the some 450 active MSGs, we are reason-- 
ably confident that 10*6 percent is an accurate estimate of the currenL 
low level of man-machine and manual (or free-^form) activity. 

Multiple choices were allowed; that ia^ a respondent could call 
his MSG bothj ^ay, a model and a man-machine exercise, However, only 
26 indicated a secondary choice^ and of that subseti''l9 chose slinula- 
tion, 3 "other J " and 2 each models or mathematical analysis* No one 
selected as a secondary choice man-machine or manual game. 

Concept and Questions 

The Important concept of purpose is inherently complex. Accord- 
ingly, our probings were detailed and intentionally redundant, Re^ 
spondents were asked to describe in their own words the major stated 
purpose of their MSG and to cite two specific questions or operational 
problems that the MSG had been used to answer. Nextj they were asked 
: to check off appropriate characterizations of purpose from a list of 

eight categories J one of which was "other—specify." We also asked 
them to indicate their confidence In that response^ from, low to abso^ 
lutely certain. Several questions later, purpose was brought up again, 
with reference to who had initiated the work, what he wanted, and how 
well he specified his wishes. ri.nd finally, several questions later, we 
asked simple yes-no questions about whether the MSG was initially de- 
signed for experimental or educational purposes. The results of this 
repeated probing are revealing, especially when tabulated with other 
descriptive categories such as who paid for, built, or used the MSG, 
at what cost, and so forth. 

Results 

* _ 

Description , Primary purposes are shown in Pig. 2. No respondent 

For a theoretical examination of purposes, see Martin Shubik, On 
the Scope of Gamfng, The Rand Corporation, P-4608, March 1971| repub- 
lished in ManagemBnt SaisnaQ^ Vol. 18, No* 5, January 1972, Part 2 
^ pp. P-20-P-36. 

EKLC 
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Imary s^afed purpose of MSG (varfable PURP 1) 



Indicfltad that his MSG was intended primarily for military-political- 
oconomlc and training/education purposes. In fact, only one MSG was 
listed under political-diplomatic-military (abbreviated PDM) /interna- 
tional relations, Given present-^day concern over strategic arms limi-^ 
tation, perpetual Middle East tensions, and the expansion of Slno- 
American relations, that is a starLllng finding. Even whan secondary 
and tertiary purposes are taken into account, the findings are not much 
different. Some 61 or 46,2 percent of the MSGs had only one purpose, 
and of the remainder, one each had secondary purposes of PDM/interna-^ 
tional relations and training/education. Of the ^BGs with three pur-- 
poses (36 or 27,3 percent), two were milltary-^politlcal-economic and 
one each was PDM/international relations and training/education. Re- 
search/methodology received similar scant attention, It was the pri- 
mary purpose of 2 MSGs, the secondary purpose of 5, and the tertiary 
purpose of 10, 

By far the greatest emphasis in current DOD activity is on all- 
romputer, technical evaluations (59 or 44,7 percent), force-structure 
analysis (38 or 28,8 percent), and doctrinal evaluation (24 or 18,2 
percent)* When it comes to secondary purposei the mi% changes. Of the 
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7J MSGs With moru than owe stattad purpuKij, H wctt? inl:fe]ndt3d t(' Hiudv 
dactrine, 28 for fcjrce striicturUj and only one for Luchnical t^valuution. 
At the tertiary lev^l, with 36 rupurttid, 17 were for f orce-^n rue tur>/ 
analyHis, lOwuro for research and mtithudnhigi cal developm^nu , and t !u' 
remainder were sprinkled among uther purposes. 

Technical evaluation turns out to mean weapon Hystem evaluatinn 
when one reads tlie written accounts of each MSG's major stated purpose, 
and eKamples of use. Such act ivi ties are appropriate enough , but un i y 
if the data are valid, are available for scrutiny by responsible ptir-^ 
suuB, and if scientific criteria and procedures such as replication, 
external review, and documentation are followed. If such rigor is not 
assumed^ one suspects that much effort Is going into building uneva]- 
uated MSGs that suppyrt apeclalized points of view* The scarcity of 
attention given research^ trainings and more-or--less "political" mat- 
tars could lead one to believe that the DOD is modeling problems that 
are easily quantified and are well enough understood that no new theo- 
retical research is needed to explain them* If so, tlien eitlier the 
problems confronting the 1)01) are in fact being managed in a rigorous ^ 
scientific fasliion, or, as appears to be the case, intangible* intract^ 
able, or "soft" issues are largely being overlooked. 

Respondents were generally highly confident or absolutely certain 
about their MSG's purpose (105 or 79.5 percent),'^ However, 27 or 20,5 
percent either had low to moderate confidence in their answers or did 
not res^pond at all. Considering the composition of the respondent 
population and tlie inquiry intent and sponsorship , that figure seems 
mordinately high. If these people do not know wliat their models are 
supposed to doj who does? 

Initiators of MSG developinent , as shown in Fig, 3, strongly favored 
analysis and diagnosis as a primary purpose (108 or 81,8 percent). Far 
fewer favored operations (14), research and theory development (7), ex^ 
perimentation (1), or teaching-straining (2), That finding is consistent 

Confidence levels, the degree of the respondent's certainty about 
an answers were required on most questions. The categories ranged over 
low, noderate, high, to absolutely certain, assigned values 1 through 
4, respectively. 
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Fig, 3 — MSG iniHator's primary purpose ( variable INPURPl) 

with the heavy mphasis on technical evaluation and force-structure 
analysis already obierved. When asked how tightly funding sources 
specified their intentions or purposes, 77 respondents replied with 
either high or abaolute certainty that 33 or 25.0 percent of the sample 
were tightly specified , that iSj the funding source defined well what 
it wanted for its money. In 26 or 19 ,7 percent of the cases ^ however^ 
the respondent either did not know or did not respond to the question. 
Those replies are shown in Fig, 4, 
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A full 114 or 85,4 percent of the MSGs were not Intended for ex-- 
perlmental purposes^ and 121 or 91.7 percent were not intended to be 
used for educational purposes. The saniple may be tentatively charac- 
terized as operational machine models and simulations used for technic- 
cal evaluations and f orce^structure analyses. Research, experimenta- 
tion, and training and education are all of considerably less importance 
in this sample. 

Trends , Fundamental purposes have shifted somewhat for MSGs Ini- 
tiated since 1966, Of all MSGs produced, technical evaluations have 
declined somewhat ^ from 50.0 percent for 1966^67 to 38,9 percent for 
1971-72; force-structure analyses have increased from 13,6 percent to 
38,9 percent; and doctrine evaluations have remained rather gteady at 
about 22,0 percent of the total. 

Since 1966, not a single one of these MSGs was initiated for 
teaching and training purposes, Analysis and diagnosis as a purpose 
has held steady at about 85 percent of the total initiated, 

Ralationship to Other Descrlptora . lespondents knew in general 

what their MSGs were supposed to do, as indicated when primary purpose 

was cross^tabulated against the respondent's confidance in his reply. 

Some 89,5 percent were either highly or absolutely certain about purpose; 

however p 10 of the 38 force-structure MSG respondents indicated low or 

2 

middle confidence or did not bother to answer at all [X" p < *024], 
The reason is not clear, but a possibla eKplanation is that force-struc» 
ture MSGs are not well documented or are "one-shot'* affairs whose 
initial purpose is no longer remembered by the organizations responsible 
for them. We shall reexmine that hypothesis later. 

Looking at the MSG funding source Indicates that the Navy concen^ 
trates heavily on technical evaluation, 26 out of 33 or 78,8 percent; 
the Air Force divldea between iorce-structure analysis^ 12/23 or 52,2 
percent, and technical evaluation, 10/23 or 43*5 percent; the Ar^ is 
evenly divided between technical evaluation^ 18/56 or 32,1 percent, 
doctrine evaluation, 16/56 or 28,6 percent , and force-structure analysia, 
16/56 or 28,6 percent; and 5 of the Joint Chiefs of Staff's entries 

were on force structure and one was on PDM/lnternational relationi 

__2 

[X p < .001]. No direct Department of State or National Security Council 



funding was noted, reinforcing an t^ar I i fir obHorvatian about the H/ifiple^K 
apparent dt^arth of explicitly political or diplomatic substance* 

I'ISG initiator cross- tabula tud with purpOHL^ requlr^fe^ snmo expl.nia** 
tioHi Models-builders or resGarcliers may n^A individuals prupusti MSC 
construction to an appropriate funding source * That dif^r-rs from Ln-= 
hjuse workj which is limited either by th^i tr/tmtual users or by a non-- 
user managiment source acting for the users. An agency outgide aii or-- 
ganization may request an organization to build an MSG for its use, or 
an agency my request a different organization both to build and use 
an MSG for the initiator's ovm purposes. Individual researchers ini- 
tiated 24 or 18.2 percent of all 132 respondent MSGSj and of those there 
was a fairly even division between technical, doctrinal s and force=struc^ 
ture evaluation, and analytical purposes. Of the 40 MSGs Initiated by 
in-house users , 18 or 45,0 percent were for technical evaluation. Thatj 
plus those initiated by users external to the building agency^ ncccnints 
for 34/59 or 57.6 percent of all the technical-devaluation MSGs, ThuSj 
it appears that users tend to initiate their own technical^'evaluation 
MSGS [X" p < ,011]. 

The initiator's purpose, whether teaching, analysis^ operations, 
experimentation, or research * when tabulated against the MSG\s purpose 
or category of intended use, shows that the penchant to do analysis 
and diagnosis runs evenly through all MSG purposes. For technical 
evaluation, 50/59 or 84,9 percent are intended for analysis and diag- 
nosis; for doctrine evaluatlonj it is 21/24 or 87,5 percent; and for 

2 

force structure, it is 30/38 or 78,9 percent [X p < .001], The one 

surprising finding is that when an i. iator intended research and 

theory development (7/132 of 5,3 perL ^^nt) , it resulted in M_SGs whose 

primary purpose was technical evaluation (4/7) , force-structure analy^ 

sis (2/7) J or doctrine evaluation (1/7) , not the development of research 
2 

or methodology [X p < ,001] # It appears that even when one has a serl^ 
ous research problem, the resulting MSG Is not reported as being pri- 
marily geared to research. As for the two MSGs whose prime purpose was 

listed as research and tnethodology , initiators intended analysis, diag^ 

_ 2 

nosis, and experimentation, not- research per se [X p < ,001], 



Twij duscripCivy claHHtiH uf IninmiaLion, the ease of transt^r of 
MSn Dporritions from am? aite to anorher and the extent of documenta- 
Liun, provide insight into the adequacy of scientific controls, 

Technical=evaluation MSC;^ do not transfer at all in 15*3 percent of 
thL^ 59 cases; transfer with gryat tixpense in tiim and technical talent 
in ^2.0 perciint of the cases, and are easily transferable in but 8 of 
the caseSj or 13*6 percent of the time. Tlie relationship between pur- 
pOHtA and c^asi? of transfer warf not statistically signif iaaiic * so we are 
unable to place too much confidence in what the descriptor means, How- 
evur, tlie tabulation of purpose versus extent of documentation was 

9 

higlily significant [X" p ■ .OOJ] and suggestive. By the respondent's 
own assessment of documentations which we found to be ovcrgenerous 
in several cases where documentation was probed in detail, 15 of the 
technical-devaluation ^BGs had either weak, poor, uneven, unavailable^ 
or unknown documentation* Of the 38 f orce^structure MSGs, 9 fell into 
those categories i That information is not conclusive ^ however ^ because 
rougtiLy eq lal numbers in each category responded that their MSG had 
excellvn^! J very goods or average documentation. While the distributions 
are flat, the fact that 25.4 percent of the technical-evaluation and 
23.6 percent of the force-structure MSGs were voluntarily acknowledged 
to have weak or worse documentation seems important* 

Cost data were broken down into coded categories and ware tabulated 
against purpose. Iranediately one is struck by the fact that 25 or 18.9 
percent of the respondents were unable to supply any information what- 
soever on costs* Of the remainder, 50 MSGs cost $100*000 or less, 22 
cost $100,000-$249,000, and 17 cost $250, 000-$500 ,000. Of the 18 
MSGs costing more than ISOOjOOOp 6 were technical evaluations^ 5 

were doctrine evaluations , 5 were force-structure studiesj and 1 was 

2 

for political-diplomatlc-inilltary purposes [X p < i012]. Later we 
shall take up cost considerations in more detail* 

PRODUCT ION 

Under "purposes several basic questions were asked about the 
MSG-s production. It is an important topic that warrants analysis by 
itself, so we later posed more detailsd questions^ such as how many 



predeceasors and spinoffs are traceable to the MSG; how long it was 
under development before being used; how many man-years were taken to 
build it; and where the data came from to specify it* Another set of 
questions asked respondents to describe and characterize the MSGs that 
resulted from tha production process. 

Concepts and QueBtlons 

While some production questions, such as what MSGs preceded or fol 
lowed the investigated MSG, are straightforward and easily answered, 
others are not. 

When discussing development t±mB^ for example, it is difficult to 
ba both precise and accurate. We refined the qusstlon to include the 
elapsed time between the decision to build a given MSG and its first 
production run, thereby grossly underestimating in those cases when an 
MSG continues to be developed even while it is being used. Another 
illustration of conceptual difficultias is the estimation of human re^ 
source expenditures. One must distinguish between total man-years j 
professional Mn'-yearSj and programmer man-years to understand what 
kinds of talent have been involved in MSG construction. 

To understand the sample better, several descriptive questions 
about scenarios p mathematical difficulty, timing, levels of resolution, 
use of random events^ and supporting data were asked. Taken together, 
they provide valuable Insights into current production practices * 

Results 

Description ^ About three^f ourths of the sample had at least one 
direct parent or antecedent MSG, indicating the cumulative, continuous 
natura of much of the work. While the sample was biased to include 
many active MSGs, there were 45 cases where a spinoff or distinct prog^ 
any was reported to have been developed* These activities are clearly 
not discrete eyants but represent a continuous, ongoing process of ini- 
tiation, production, and use. 

Alternative procedures or methods to the one chosen were elicited 
by asking the respondents to imagine achieving the objectives of the 
^SG by different means. As shown in Fig, 5, analysis was the dominant 
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Fig,5~ Best alternaMve method to one used (variable ALTPRO ) ) 

alternative, reinforcing an earlier observation on the widespread use 
and acceptance of the term "analysis.'' The face that 22.0 percent of 
the respondents believed that no alternatives eKisted supports the view 
that much of this activity is of a "last-resort" methodological variety! 
many of the problems are not tractable by other means. 

MSGS were constructed largely by the armed forces themselves, 59 
or 44.7 percent of the entire sample. For-profit organizations built 
37 or 28.7 percent; not-f or-prof its contributed 29 or 22.0 percent; 
and universities accounted for 7 or 5,3 percent. 

Development time^ the elapsed time between initiation and the first 
production run, varies widely between less than three months to more 
than 42 months. The distribution is shown in Fig, 6, Since 63.6 per- 
cent were developed in 18 months- or less, one might conclude that de-- 
cislonmaking lead times are most likely to run under two years. It 
took two to five man-years to develop 31.8 percent of the sampled 
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MSGS. Less dominant categories are depleted in Fig. 7. Professional 
man-years and total man-years are nearly equivalent, suggesting that 
most development activity was carried out by professionals themselves. 
Indeed, written profiles of profesaional teams support this point 
nicely. About one computer programmer man-year per MSG was Che norm. 

Our next set of questions dealt with a characterization of the 
MSGS and their data bases. A quick summary of the findings revealed 
that 54 percent used numerical scenarios only. Judging from the analy- 
sis and written replies to the questionnaire, little attention appears 
CO be paid to intangible assumptions or to qualitative factors in the 
studies . 
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In spite of a popular belief that "advariced mathematical methods" 
are taking over, the level of mathematical sophistication required to 
work with the MSGs was regarded as high ("requires an advanced degree 
to interpret the output") in only 8 percent of the cases* 

Most ffiGSj 52 percent j had as their t;ime setting either an unspeci- 
fied time or the future, 

Oi: all the MSGs^ approximately 45 percent were directed toward the 
study of combat at the engagement level, 10 percent at the battle levels 
11 percent at the campaign level j and 14 percent at the level of war. 

The level of temporal resolution was rather small. Fully 79 or 
66p8 percent were detailed down to either minutes or seconds. Similarly^ 
spatial resolution was also fine-grained; about 35 percent were based 
on meters, and some 27 percent were detailed at the kilometer level* 

The sample was split about SO/50 on the use of random elements , 

We were surprised, given the quantitative and qualitative advances 
in computer languages, that so many MSGs were still written In FORTRAN— 
more than 80 percent. The remainder were spread widely among SIMSCRIPT, 
ALGOL, ASSEMBLY, COBOL, GPSS, and several others. 
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On the question of MSG siEe, we asked for the approximate number 
of computer inscructions contained in the model itself, less any data 
inputs. On the average, they are either very large (36 or 27.3 percent 
iiad over 10,000 instructions) or moderately small (24 or 18.2 percent 
had 2^000 or less); 45 or 44,0 percent contained less than 4,000, or 
about two boxes of IBM cards. 

In spite of our difficulty in getting information about these MSGs, 
72 percent were unclassified as long as data input valufas were not at- 
tached. With input data, only 12 percent were still unclassified. Data 
are classified predominantly at the Secret level. Thus, there is little 
reason for classification to prevent most of the sampled MSGs, at least, 
from being made available for scientific scrutiny. 

There is an indication that data-source and validity problems are 
acute. Careful gathering of field- test data or eKperimantal informa- 
tion tends to be expensive and time-consuming, especially the prepara- 
tion of planning factora; few of these MSGs benefited from such measures. 
The modal procedure is to have other military agencies supply data with- 
out any follow-up or cross-checking by the user agency. In less than 
30 percent of the replies was there any Indication that additional ef- 
fort had been made to check the validity of the data. 

Tre^^. Of MSGS developed since 1966, development time has fluc- 
tuated so that no clear trends are apparent. There is a slight trend 
toward fewer technical evaluations and more force-structure analysis; 
doctrine evaluations' have been relatively stable. 

The technical characteristics of MSGg are all quite stable. That 
is to say, there are no discernible changes in the proportion of nu- 
merical versus verbal scenarios being used, in the lever of mathemati- 
cal sophistication being "built into" the MSGs, in the level of temporal 
resolution, or in the split between those using and those not using 
stochastic or random elements* 

Relationship to Other Descriptors . Of the 59 MSGs constructed in-- 
house, 31 were for the purpose of technical evaluation , 17 were for 
forced-structure analysis, and 8 were doctrine evaluations. Of those 
built by for-profit organizations, 11 each were technical evaluations 
and force-structure analyseSp and 8 were doctrine evaluations, Not- 
for-proflts built 11/29 or 37*9 percent for technical evaluation, 10/29 
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or 34.5 percent for f orce^structure analysis, and 8/29 or 27,6 percent 
for doGtrine evaluation. Universities concentrated 6 of their 7 MSGs 

on technical evaluation, Of the 2 research MSGs, 1 was done by a unl- 

2 

varsity and 1 by the armed forces [X p < .007]. 

Multiple authorship or participation in the conatruction phase wag 

noted in 17 instances, Not--f or-prof its did 11 of thenij mostly on a 

piecemeal or consultative basis, and 8 of those were for technical aval 

uatlon purposes. The remaining 6 inultiple--auchor MSGs resulted when 

several agencies of the same armed force contributed significantly to 

2 

one final product [X p < ^016], It appears that the not-for-profit 
firms are Dccasionally utilized to back up In^house, anned forces con= 
struction activities. Given the scope, magnitude, and cost of many of 
these MSGs, the extent of multiple authorship is less than we had ex-^ 
pected. In fact, it may be a manifestation of what appears to be a 
harmful compar tmentalizaCion and overspecialization in activity^ 

When the military service responsible for an MSG is tabulated 
against a range of construction-related descriptive variables, the fol- 
lowing patterns emerge: 

Construation and purpoas: The Army Is rather evenly split between 
the three purpoges of technical evaluation, doctrine evaluation, and 
forceps tructure analysis; the Air Force is doing mora force-structure 
analysis, 50*0 percent^ and technical evaluation, 42,3 percent, than 
doctrine eyaluation, 7.7 percent; and the Navy is largely constructing 
technical evaluationi, 77.8 percent, to the exclusion of other types 
[X^ p < .0001]. 

Initiaiion: Navy MSGs are initiated either by external users or 

by the builders themselves to a far greater extent than in the other 

services. The Army accounts for 11 of the 1/ or 64,7 percent of the 

external, non-user initiations. No particular pattern is discernible 

for the Air Force; that is, initiation comas from a variety of sources 
2 

[X~ p < .03], There is also no observable pattern in Air Force speci- 
fication of the purposes of its MSGs. The Army, on the other hand, ac- 
counts for two- thirds of those that are tightly specified. The bulk of 
Navy work was noted as b^lng "-^nderately'' specified [X^ p < ,002], 



^'^>:ntrucftion site: While not signit iiNiiu ly related [X^ p ■ ,1^^. 
n.s.], on a percentage basis the Air Fnrctf does mnre of its own wnrk 
in^house than do the other services; the Anny uses for--pruf its more 
than expected and more than the ocliers; nnd the? Navy's involvement witli 
Johns liopkins University makes it the dominant university user, 

Developffwnt: As noted, abo^ut two-'thirds of the sampled MSGs were 
developed in 18 months or less. Tlie Army accounted for 21 of the 36 
MSGs built in 6 months or less. Navy construction apparently normally 
takes from 6 months to a year, wlille no distinct pattern is evident for 
Air Force construction. Likewise, at the other end of the scales no 
one service stands out as taking particularly long to develop its MSGs 
[X^ p < .05]. 

Data 8om^aM and validity: Navy data, as indicated by the respon- 
dents, are most likely to have been cross-checked, field- tested ^ or ex-- 
perlmentally derived, 27/36 or 75.5 percent. For the other services, 
It was about a ttoss'-up as to whether data were checked or not, MSGs 
built at or for the Joint Chiefs of Staff were about three times as 
likely not to have their data checked as were all others in the sample; 
in fact. In 10 of 11 "other DOD'* ^^SGs the data were not cross-^checked, 
were unknown, or no response was given. Validation procedures were not 
significantly service-specific, although no less than 38 or 28.8 percent 
of the entire sample reported that validation was not undertaken, was 
unknown, or gave no response, 

Data type : We asked what predominant type of data were usedj 

whether hard, moderate, soft, or combinations . EKamples of each of 

three basic types were provided. Only 8 used mainly soft data or soft 

data in combination with numbers of greater certainty, a point already 

noted, Generally^ Air Force MSGs used hard data to a greater extent 

2 

than did those of the Navy^ or Army [X p < .01], 

ConBtruation and B&aunty alassifiaation: If an MSG without data 
Inputs was classified at all, and about one-fourth were, it was most 
likely at the Confidential (17/37 or 46,0 percent) or For Official Use 
Only (9/37 or 24,3 percent) level. More Army MSGs are unclassified 

(81.4 percent) than Air Force (69,2 percent) or Navy (64.0 percent) 

2 - 

[X p < ,01], When data inputs are added, a full 85.6 percent of the 



tiiif ire sample becomes classified, mastly at Che Secrec level (78 or 

59.1 perconc). On a perciirUage basis, Che Air Force at 92.3 percent 

UnH more clossified ?\SGs than expected on the average for the entire 

saiiiplej and more than either tliy Navy or thu Army [X" p < ,002]. 

.7;;^:? The hinindal distribution noted for the whole sample 

lield for the Indtviduril services with only minor variations; the Navy 

at 33 percent, on a straight percentage basis, had more MSGs in the 

1 

largest size category (: 10,000 inHtructions) than did either the Army 

(28.8 percent) or the Air Force (19.2 percent). Most Air Force entries 

were in the 1,000-2,000 instruction range, 9/26 or 34,6 percent. If 

Army MSQs were not large, and 1.7/59 were notj they were in either the 

2,000-4,000 or 4,000-6,000 range (both 8/59 or 13,6 percent) [X^ p < .001]. 

Lt::i?elL^ of M:Jil vcihjUiLiun: The Navy is building more of its MSGs 

at a finer level of temporal detail than are the other serviGes; all are 

cast either in seconds (22/36 or 61,1 percent) or minutes (8/36 or 22.2 

percent), with the remaining 6 not answering [X p < ,001] ^ The Army 

builds most of its MSGs at the level of spatial detail of the kilometer j 

the Navy at the meter level, and the Air. Force shows no clear preference. 

In keeping with the fine grain of much of the sample, most MSGs were 

cast at the engagement level. Battles, campaigns, and wars, increasingly 

large levels of resolution for military aetlvity, are mainly the province 

of the Army, which accounts respectively for 85,7 percent, 40,0 percent, 

2 

and 61.1 percent of the activity in each [X p < ,001], 

When >ISG category was tabulated against the ratio of model time to 

real time, a technical concept measuring the extent of time compression 

or expansion, a full 30,3 percent of the respondents either did not know 

or failed to answer. Of the remainder, if an MSG compressed time ex- 

tremely, it was probably a simulation; real-time representations were 

fairly evenly divided among the categories; and expanded MSGs were mail- 

2 

machine exercises or models [X p < .03], 

The smallest MSGs, those having fewer than 1,000 computer instruc« 

tions, were models about 73 percent of the time. More than half of the 

man-machine exercises were in the largest size category, 54.5 percent, 

while only 33 percent of the simulations and 21 percent of the models 
2 

were as large [X p < ,02], We looked at the incidence of intangible 
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assumptions and MSG size and found that for tlie largest categorv, In^ 

tangible assumptions were made nearly 20.0 percent more than would he 

2 

expected for the whole sample [X p < .001]. 
OPEMTIONS 

Questions relatad to MSG operations dealt with two broad matters: 
scientific standards and technical-procedural issues* 

Concepts and Questions 

Of major concern was what testing and professional controls were 
employed to insure MSG fidelltyp Questions such as whether sensitivity 
testing was carried out^ whether the operations of a given MSG could be 
transferred to a comparable location elsewhere * and whether an indepen- 
dent professional review had been carried out are examples of our opera'- 
tional control questions. Questions about the MSG's need for special 
facilities, languages, or dccumentation pertained more to techniques 
and hardware* 

Results 

Description , Sensitivity analysis is an important operational 
controls especially when the number of variables is large and the model 
complex. One must find out about an MSG's behavior as sets of input 
parameters are altered in Interesting ways. Without sensitivity analy- 
sis, one cannot know much about the MSG-s perf ormance. In our sample, 
45 percent indicated that their MSG had not been sensitivity-tested. 
Granted^ such testing is generally expensive and time-consumingj but 
it is essential to a determination of the MSG's usefulness and validity. 

We have already touched on the topic of data validity. Probing 
a bit more, we found that about 14 percent used several generally ac-- 
ceptable and commendable means or checking on the accuracy and quality 
of input data. In about 28 percent of the cases, less thorough measures 
were taken, because questions remained about the precision of some num- 
bers actyally being used. For 30 percent ^ the written commentary com" 
bined with catagorical assessments indicated that the issue was at least 
considered and some effort made to check out the data. Data validation 
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was not attempted in 16 percant of the casas, and a full 13 percent 
either did not reply or did not know about the matter. 

Documentation is a prime control function that has received in- 
adequate attention. Figure 8 gives a clear portrayal of what appears 
to be the optimistically biased self-assessments of most respondents. 
These assessments would be more believable If they had been put to , 
and had passed » an nperatlonal test. In fact, as shown in Fig, 9^ only 
18,2 percent of the MSGs were generally transferable. The Importance 
of replication is both scientific and administrative. If a model can'- 
not be transferred^ independent reviewing is mQre-or--less precluded. 
The administrative issue is the obvious one of controlling redundancy* 
some 52 percent of the respondents indicated that they were not aware 
of any closely related MEGj and approximately 4 percent did not reply 
at all. But that is of little consequence becausa nearly half the re^ 
spondentas as depicted in Fig. 10, did not know what it would cost to 
transfer operations. 

Responses indicate that nearly half the MSGs had not been sub-- 
jected to review outaide the building-user organisation. The actual 
figure is probably higher ^ since this sample is biased toward Army 
models j which probably have benefited from the recent ad hoc Army 
Model Review Cotmnittee* s efforts to increase the outside professional 
review of MSGs* At best there is less than a 50/50 chance that any 
active model in the inventory has been reviewed. The date of most 
recent review is iuggested in Fig, 11, 

Special facilities of one variety or another were required for about 
a third of the sample. About 16* 6 percent needed dedicated computational 
systems, 13,0 percent used specialized languages or libraries^ and 2,3 
percent required special buildings or laboratory facilities. MSGs should 
be made more transferable than they arei there are few valid technical 
obstacles to replication. 

Trends . There has been a significant decline since 1966 in the 
incidence of external professional review, in spite of the recent ad hoc 
Army initiatives. Percentage data are summarized in abbreviated form . 
in Table 1, 
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Table 1 

TRENDS IN EXTERNAL PROFESSIONAL 
REVIEWING OF MSGS 



Incidence 


1966-1967 
(%) 


1971-1972 
(%) 


AX 


No review 


22.7 


50.0 


+27.3 


Review 


59.1 


38.9 


-20.2 


No responsa 


18.2 


11.1 


-7.1 



Since 1966, there has been a slight dacrease in the percantage of 
MSGS not intanded for transfer and those that are generally transferable 
It appears that the extremes are moving toward a middle category of dif- 
ficulty of transfer. 

The lack of knowledge about transfer costs has remained ateady at 
about 40 percent in the same period. In other words, a consistently 
high proportion of respondents had no idea what it would cost to repli-- 
cate their MSG elsewhere* 

The descriptive attributes for data sources , data-validation pro- 
cedures ^ extent of documentation, and incidence of sensitivity testing 
remained stable over the period of the analyaiSi 

Very little has ever been published in joufnals or books about the 
J^Gs in this smple* However, since 1966^ there has been a major per- 
centage Increase in use of the Defense Documentation Center* The pri- 
mary form of documentation has been relatively stable over time — ^'-re- 
ports only" account for 25 to 30 percent* "user manuals only," slightly 
less than 10 percent—but "user manual plus program decks" as a category 
increased from 13 to 28 percent in the period. 

Relationship to Other Descriptors , Here we consider the linkage 
of MSG operational characteristics and the military service responsible 
for the MSG, and the effects of external professional reviewing on the 
sampled population. 

No significant relationships were observed between the responsible 
service and any of the following operations-oriented descriptorii ease 
of transfer, cost of transfer, incidence of sensitivity testing, and 
speciallged faciiities. 
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The amount of external professional reviewing initiated by each 
service was significant. Based on the average for the entire lample 
(see Table 2), Air Force MSGs are least likely to have benefited from 
external reviewing* Adding the number of no^responses to the number 
of MSGs not reviewed reveals that over 80 percent of the sampled Air 
Force activity was evidently not subjected to external serutinyt The 
data also indicate both the extent of the Army Models Review Committee 
recent ef forte and the low rate of review for "other DOD" MSGs, 



Table 2 

EXTERNAL PROFESSIONAL REVIEWINQ, BY SERVICE 





MSGS 










Not 


MSGS. 


No 




Servica 


Reviewed 


Revlawed 


Rasponse 


Total 


USA 


15 


36 


8 


59 


USAF 


18 


5 


3 


26 


USN 


13 ' 


21 


2 


36 


Other DOD 


7 


3 


1 


11 


Total 


53 


65 


14 


132 



[X^ p < ,003] 



When semrice is tabulated againit the date of lateit professlorial 
review j we aee that 25.4 percent of the Amy's entries were reviewed 
within the six monthg prior to the cutoff date of this iurvey — around 
October 1971— a significantly higher percentage than those of either 
the Navy or Air Force* 

Of the 5 Air Force entries that were given external professional 
review s 3 were last looked at between one and two years agOj 1 more 
than three years ago, and 1 less than six months prior to the survey's 
cutoff date* The portion of Navy MSGs reviewed is consistent over time 
[X^ p < ,05], 

The Incidence of external professional review relates to other 
characteristics of the sample as well, 

Willie MSGs used for analysis and diagnosis ^ teaching and training, 
or operationi eteod about a 50/50 chance of being reviewed, those with 
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research applications did not fare so wull. Of the 7 research nr oy- 
perimentation MSGSj only 1 had had a professional review. It is curi- 
ous Indeed that the few axplicicly scientific examples were nuL sub-- 
jected to this rudimentary scientific control procedure, 

Professional reviewing was not signif iiianrly related to transfer- 
abllityj transfer cost, size of the MSG, total cost to build, extent 
of documentation J type of data mainly usadj or the incidence of sensi^ 
tivlcy testing. 

There Is a significant telatlonshlp between the data sources of 
an MSG and Its later receiving externa] review. If an MSG (1) under- 
went checking of data, (2) derived from field exercises, or (3) used 

data from a variety of experimental and operational sources, the chances 

2 

were doubled that a professional levlew would be done [X p < ,005], 
Likewiae, if data validation procedures, as described in written com- 
mentary ^ were highly or moderately rigorous (and 55 or 41,6 percent 
were so classed) , the chancei of subsequent professional review were 

also nearly double those of MSGs having less rigorously validated data 
2 

[X p < ,02], This suggests that concern for rigorous design and pro- 
duction carries over into operational control procedures as well. 

Unfortunately J that suggestion does not hold for sensitivity test-- 
Ing, which was not clearly related to professional reviewing. Curiously, 
sensitivity testing was strongly related Co MSG size. MSGs having less 
than 4,000 instructions, i,e., the relatively smaller entries, were 
about twice as likely not to have had some sensitivity testing as the 
larger MSGs having 4,000-10,000 instructions. The largest size cate-- 
gory (greater than 10,000 instructions) was evenly split on the ques- 
tion [X^ p < ,001], It Is not readily explicable that the procedure 
is carried out less frequently than expected for the smaller and prob- 
ably more readily and inexpensively tested MSGs. 

Documentation and the incidence of professional reviewing were 
related in several interesting ways, 

MSGs that had been reviewed were twice as likely as those not re- 
viewed to have their documantation located in the Defense Documentation 
Center, and 65 MSGs in all received some sort of external professional 
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review.. Wlienj for this subset, assessments of data^validaLion proca-- 
du res ar^ tabulated agains t the locat ion of documentation , a highly 
significant: finding emerges. Of the 34 MSGs thought to have high- and 
moderates-quality data validation, no less than 19 have documentation 
that is not generally accessible, being either proprietary/noc-^for^ 
profit organization (3) , proprietary/contact author (3) , proprlatary 

(classified) /contact author (10), unknown/not available (2), or out--of-^ 

2 - ' 

print (1) [X p < .001]. Once more we begin to take a measure of the 

extent of isolation of much current activity. Of cha 35 MSGs whose 
documencatlon was publics located in tlie Defense Documentation Centeri 
only 15 were credited with lilgh or moderate data-validation procedures , 
significantly fewer than those having inaccessible or limited-access 
documenCation. This finding is interesting enough to warrant repro- 
ducing Table 3 in full. 

Table 3 

CROSS^TABULATION OF DOCUMENTATION LOCATION AND QUALrTY OF DATA VALIDATinN 
FOR MSGs THAT WERE EXTERNALLY REVIEWED 

(N - 65) 



Documentation 
Location 
(Variabla DOCLOCl) 




Quality of Data Validation (Variable DATAV) 


Unknown 


High 


Moderate 


Weak 


Not 
Done 


No Re- 
sponse 


Total (%) 


Unknown 


1 


1 


1 


2 


1 


0 


6 (9,2) 


Out of print 


0 


1 


0 


2 


0 


0 


3 (4,6) 


Proprietary; not-- 
for-profit or^ 
ganlEaClon 


0 


0 


3 


0 


0 


0 


3 (4,6) 


Proprietary^ con-= 
tact author 


2 


1 


2 


0 


1 


0 


6 (9.2) 


Proprietary 
(clmsilfiad) \ 
contact author 


0 


4 


6 


1 


0 


0 


11 (16,9) 


Public; in DDC 


2 


3 


12 


8 


9 


1 


35 (53.8) 


No reiponse 


0 


0 


0 


0 


0 


1 ■ 


1 (1*5) 


Total 
(%) 


5 

(7.7) 


10 
(15,4) 


24 
(36.9) 


13 
(20.0) 


11 
(16.9) 


2 

(3.1) 


65 

(100,0) 



^ NOTEj Raw chi square « 61,83, with 30 deg freedom. Significance « 0.0005. 
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USE 

The question of use is a difficult one. There ie little consensus 
on answers to ^ny of the questions we raised and little evidence that 
they are even being discussed. Our Intention in this section and the 
one on coat that follows is primarily to initiate discussion on these 
neglected matters, and secondarily to take a reading on what appears 
to be the present state of affairs, 

Concepti^ and Questions 

Besidei asking whether an MSG's use corresponded to the major pur- 
pose for which It was constructed^ we created other measures of use* 

One standard is the market measure'--Will the >EG sell| is its fund-^ 
Ing regularly renewed? Not exactly an ideal erlterion, it provides a 
crude pragmatic indicacion of the client *s satisfaction and willingness 
to buy and use more of the same. 

Two operational^ questions approached the matter slightly differ- 
ently. The question I "How many briefings were given, based on the re- 
sults produced by the MSG?'* may give one an idea of how much stock a 
user places in a particular MSG. If no briefings resulted, that tells 
something about how key partlcipanta assess a given enterprlie. The 
corollary question, "How many times is the MSG referred to in making 
specific operational decisions?" further refines and clarifies this 
measure. To push it to* something like an ideal criterion, one should 
interview all relevant participants to determine precisely how an MSG 
related to a sp°eciflc deciiioni who advocated Itj who voiced reserva- 
tions, and what official rationalizations derived from it. We had 
neither the time nor other resources to do that. The questions of who 
initiated the MSG and for what purpose come to bear at this point to 
dramatize an MSG-s use. Straightforward questions such as, "Is the 
MSG active or not?" and "How often is it operated?" provide further 
important information. 

Operations and use overlap when considering what kinds of documen- 
tation are located where ^ how easy it is to transfer use from one site 
to another, and the rate at which and reasons why an MSG becomes obso- 
lescent » 
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Description . Primary caCegories of actual use are shomi In Fig. 
12. The emphasii among initiators on the use of MSGs for analysis and 
diagnosis Is reflected nearly exactly in the way respondGnts indicated 
l^Gs are being used. The other possible primary utilization categories 
taken together account for less than 22.0 percent of the sample. Re- 
spondents indicated that 42 or 31.8 percent of the MSGs have secondary 
uses and that 13 or 9,8 percent have tertiary uses. Of those with sec- 
ondary uses, 30 are used for operations, 5 for analysia and diagnosis ^ 
4 for research and theory development, and 3 for experimentation. 
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Unknown 

1(4)3.0% 
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m (6) 4.5% 

Research and theory 

3(1) 0.8% 

No reiponse 
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Fig, 12 — Respondents* prfmary clasiification of MSG 
(variable USE 1 ) 



use 



About 80 percent of the reapondents were highly or absolutely 
certain about how their I^Gs are being used. 

Queries about genealogy (discussed on p. 22, above) revealed that 
nearly three-fourths of the sample have a direct antecedent and about 
one-^thlrd have already spawned offspring, The crude market tneasure 
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of utiUzacion tells us that business is good and clients are suffi- 
ciently contenEed to keep funuH flowing. 

Several questions were asked about briefings. For some mndels 
thfi number of briefings may not be a good measure, but for games and 
simulations it usually is. Frequently a briefing may he no more than 

superficial performance, but at least it indicates Chat someone con- 
Hidered the work interesting enough to produce. To the simplest ques- 
tion about the total number of briefings based on the particular MSG 
{sfee Fig. 13), 11.4 percent of the respondents replied "none," and 
^2.4 percent simply did not know. 
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Fig, 13— Tofal number of briefings hmed on MSGs 
(variabia BRIEF) 

Respondents were asked for written cemmentary on the level (or- 
ganization and personnel) and purpose (decisions dependent on MSG) of 



the briefings ihac were given. Fht^ 61 MSGs fur which answers w^rc 
given wuro commonly briefed to the proj cu:t; s ta f f , to an immed iat e su- 
perior in the organ izaEional hierarchy, or occasionaily to flag=ievdl 
officers and their civilian count er par c at the top command levels of 
the services. The^ purposes of thosu brUfings were not rtipurtivd con= 
sistently enough to allow simple coding and categorization ^ su verbal 
summary abstracts were recorded for eacli MSG by name. Examples of com- 
mon replies include the following i used for input to oth.r studies 
(SEANITEOPS) , strategic posture choices (NEWCON) , information on F-15 
performance (TAG AVENGER), no decisions based on inodel (AREA DOMINATION 
II), and selection of weapons for DLGN-25 (AAWSEM) . 

Generally, briefings appear to have been generated for the benefit 
of one or a few colonels or generals (or their naval and civilian 
counterparts) who occupy studies and analysis roles in their respective 
services* What further use these audiences made of the briefing Infor- 
mation is not determinable from the questionnaire answers and would re^ 
quire extensive personal interviewing to resolve. 

Let us reiterate that nearly half the MSGs surveyed had not pro^ 
duced a single briefing. 

The question of who initiates tlie development of an MSG was dis- 
cussed earlier (p, 17), As revealed in Fig, 3^ analytic and diagnostic 
purposes predominate. Comparing those data with responses in Fig* 12 
suggests that to a marked degree initiators are getting what they re- 
quested. 

The frequency of MSG opefation suggests another dimension of use, 
A distinctly bimodal distribution is evident in the data sho^'m in Fig* 
14 on the average annual frequency of MSG use, 

Llinitations on the availability of doeumentation and problems of 
transferability 5 touched on when describing operations, do little to 
promote widespread use. We asked about the availability and location 
of documentation. Public availability was concentrated in the Defense 
Documentation Center; no MSGs had documentation in tlie Library of 
Congress; and only one was listed as having been written up In journalB 
or books. Multiple sources of documentation were listed^ but only. three 
respondents checked two sources, and only two checked three separate, 
sources » 
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Ffg, 14— Average frequincy of MSG operaHon per year (variable OPFRQl) 



The cost of keeping an MSG up-to-date and, to a lesser extent j the 
average cost of runntag an MSG bear on its utilization. To get at an- 
nual updata coitij we asked for information on total dollars, profea- 
alonal man-years p and prograimner man-years expended over and above 
"normal" operating costs. For the 71 mGa for which repliei were given, 
cost data are luiraarlied In Fig, 15. It is clear that updating is not 
a trivial matter; likewise, the lampled MSGb are generally eKpenslve to 
run. Figures on the cost of a single MSG run| assmlng only minor or 
no variations in Input values, are shown in Fig. 16, 

Trends . Th^ availability and location of documentation have changed 
somewhat for mCs that have become operational since 1966. On a per= 
centage basis, more documentation is being loeated in the Defense 
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Fig. 15 — Total annual MSG update costs (in $ thousands) 
(variable CSTUP) 
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Fig, 16— Cat of e single run of MSG (in $ thousands) 
(variable CSTRUN) 



Dacunientation Center, but more alnn proprietary, classifl^ui, niiii 
available only from the author, Vvwer m^:^ hnvo documentation thai 
is proprietary, available ^frum a not-for-profit firm and propr 1 t>ta ry , 
or unclassified and available only from Liit^ juthur. 

Average annual update costs are changing; in significant waVH. 
Fewer MSGs, .n a percentage basis, aro updatud for lass than ^lO.noO 
now than in 1966 (16,7 percBnt now verHUB J6 . 4 percent then), but no 
MSGS currently fall in the $50, 000-$100 , 000 range. What has happened 
is that MSGS in the two intarmediate range^i liave increased, as sham 
in Table 4. Other trends in use are not so clearly discernible. 

Table 4 

TRENDS IN THE COST OF ANNUAL UPDATE 



Cost 
($ thousands) 


1966-1967 
(1) 


1971-1972 

m 


A% 


$ < 10 


36.4 


16.7 


-19 


7 


$10-$24 


9.1 


11.1 


+2 


0 


$25-$49 


9.1 


22 . 2 


+13. 


1 



Relatlgnshlp to Other Descriptors , Respondents were highly or 
absolutely certain about the actual use of an MSG in nearly 80 percent 
of the cases. No significant association was noted between category 
of use and level of confidence. 

The correspondence between the initiator's intended purpose and 
the actual use to which the MSG was put was strikingly consistent. If 
sonieone wanted an MSG for analysis and diagnosis, he got it 97/108 or 
89,8 percent of the time. The other 11 MSGs ended up being used pri- 
marily for teaching and training (3), operations (3), experimentation 
(1), research and theory development (1), or^ the respondent did not 
know or did not reply (3) [X^ p < .001]. Those deflections of intent 
are minor indeed. 

When the respondent's role was tabulated against his assessment of 
the quality of his MSG^s documentation, we found that funders and npon- 
sorg were the most likely to claim excellent documentation rather than 
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any other possible quality category, 7/19 or 36.8 per enC. Users in-^ 
dicatL*d uhat dQcumentatlan wns very good inore frequent!/ than other 
categories, 16/42 or 38,1 percent. Designer/builders opted for aver- 
age, 14/41 or 34,1 percent, and very good, 12/41 or 29,3 percent j in 
nearly equal measuri, Hciwuvor, care-takers indicated predominantly tliat 

dncumentation was either average, 8/18 or 44,4 percent, or poor, 4/18 

2 

or 22.2 percenc [X*^ p .00] ], Wiilta several interpretations are 
plausible. It seems that thoBC farthest removed from actual MSG use 
(funder'-sponaor) are the mo^t glowing in their assessments of document 
tation quality. 

On the average, about one- third of the respondents did not know 
whether their MSG had been briefed or not. Wien those for which no 
briefings had been given were totaled with this unknown category and 
then tabulated against responsible military service, the information 
In Table 5 was produced. The relationship is not statistically sig^ 
nificant, but it suggests that the MSGs in this sample are either not 
being used or, more likely, that people responsible for their use are 
not keeping track of how often they are used. 



Table 5 

BRIEFINGS BASED ON MSGs, BY SERVICE 



Service 


MSGS 
Not Briefed 
or Unknown 


MSGS 
Briefed 


Total 


USA 


34 


25 


59 


USAF 


12 


14 


26 


USN 


14 


22 


36 


Other DOD 


11 


0 


11 


Total 


71 


61 


132 



Annual frequency of operation and update costs are related to the 
individual services in significant and interesting ways. The bimodal 
nature of annual use holds for all services* A gi\^en MSG will, about 
25 percent of the time, be operated one to five times a year, if at 
^ all; at the other extrfime, it has about a 15 percent chance of being 
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operated in excess of one hundred times a year [X p < t05]. Annual 
updating CQgta are approKlmately the same for all aervices. For their 
part of the total sample, 30 percent of the Army's MSGs cost lass than 
$10,000, compared with 34.6 pereent of the Air Force's and 22,2 percent 
of the Navy's, The next most frequent cost range is $25 , 000-$495000, 
\^ich accounts for about 12 percent of each service's MSGs [X p < .005]* 

COSTS 

A somewhat cynical, but wise, glister of our acquaintance claims 
that gaming activities cost whatever the bookkeepers want to make them 
cost. There are many lengthy, complicated reasons validating that re- 
markj but suffice it to say that costs are deceptively hard to deter^ 
mine* 

As the work Involves many invisible costSj Dverhead costs , Jointly 
shared facilities , and jointly used products, formulating a meaningful 
costing procedure poses deep scientific problema that are far from 
being resolved. The respondents to our survey found it difficult even 
to give a simple cost description* 

Concepts and Questions 

The questionnaire contained a large number of cost questions. Be-- 
sides attempting to characterize the nature of costs using several 
plausible dinensions and categories, we were Interested in determining 
the amount of current attention to costs and the general level of 
knowledge in the profession about costs, 

Answers on all three topics—nature, attention, and knowledge — 
were disappointing. More than a third of the respondents simply did 
not answer cost questions, and of those who did, the variability of 
replies was large and the level of confidence was low. If these qual^ 
ifled prof essionalSi being asked coat questions by the U.S. General 
Accounting Office on behalf of the House Appropriations Committeei 
could not come up with some sort of figures , it is difficult to see 
who can, unless cost records are kept with the work. 



Results 

Description . Though nearly three-fourths of the MSG sample had 
at least one direct antacadent ^ 62 1 1 percent of the respondents eitht^r 
did not know or failed to respond to a question about the costs for 
these MSG families* Moreover » of thbse that anst^ered, uncerCalnty 
about the answers was high; over half had low confidence in their re- 
sponses. This suggests that cost considerations are discontinuous, 
i.e.i costs are not accumulated i even though changes in the MSG may be 
marginal from one version to the next. Costs easily become aeparated 
from substance. 

Direct funds, money formally assigned for construction purposes * 
were used by 61 or 46*2 percent of the sample. How these funds were 
distributed is shown in Fig, 17* Confidence levels were low in nearly 
half the cases. 

Total or gross costs-'-direet ^ indirect, imputed^ and unimputed---- 
are depicted In Fig* 18, The confidence level for this roughest approK-- 
imation of costs was slightly better than for the other more detailed 
cost categories but considerably worse than for other kinds of vari- 
ables, For instance^ on the classification of tffiG purpose^ 80 percent 
were either highly or absolutely certain. On total costs , only 20.4 
percent were as confident. 

Other cost categories have been, discussed in earlier sections. 
For example s man-years expended to construct an MSG and sources of funds 
were mentioned under ^'Production'^ ; transfer costs and special facilities 
under "Operations*'; and update and single^run costs under "Use,"' What 
those sections reveal only confirms our impression that (1) generally 
accepted, simple accounting definitions do not exist, (2) Insufficient 
attention is paid to costs to manage these MSGs effectively, and (3) few 
people know how much money was invested in their MSG, how much la being 
spent to operate and maintain It, and what monetary benefits are ac- 
cruing. It appears that elementary questions such as "\^hat was bought 
for how much?'* seldom get asked. 

Trends i Trends in three categories are worth considering: total, 
transfer I and update costs. The incidence of "unknown" and no responses 
for total cost information on all MSGs is steady at around 25 percent 
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Fig, 17- — ^Direcf funds for MSG construction (variable FUNDIR) 
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Fig. 18^ — Total MSG c<^tS: Direct^ indfract, imputid^ unimputed 

(variable TOTCST) 




for cho years aftar 1964, In other wurdF, a fourth of chv* rt?ypondent- 
diu--rat^ evan hoth^^r to gaess, Thu priipnrtton of MSciw in each nf 
cost ranges has"Tti^uAi;ieJ relatively stfihle over tinife?* Unknowns and lu- 
responses regarding transf er^'TO\sia_ i^UM=u^^Btid over th.*sy regarding toL.il 
costs to a5 p^rrBnt for MSfis initiniud a f feT^^-l^^fij '^'^^^i amount of no- 
responijes and unknowns regarding update costs is about st^^I^^at AO 
percent , 

Relatlonstilp to Other Descriptors . Bacause so little informatinn 
was given, fewer Htatistical tables were generated for cost than for 
other descriptive VHriables. No signiftcanc associatipns existed betwt'cn 
total costs and the incidence of professional reviewing, data sources^ 
or general classification (models simulation, man-^machlne game , etc). 

When military service was tabulated against direct costs, the data 
in Table 6 were generated. When the process was repeated for total 
costs (Table 7)^ the pattern shifted somewhat. Not only were there 



5re responses, but the distribution changed. The information in this 
ibie is not statistic; 
less worth presenting* 



0 

table is not s tatis ttcally significant [X^ p < ,09, n.s,] but neverthe- 



QFINXONS OF RESPONDENTS 

We^ asked the respondents their professional opinions about a num- 
ber of issues j including the potential usefulness of clearinghouses , 
regional centers, and external professional review boards* We built 
radundancy into the questionnaire In this regard, asking similar ques= 
tlons in different portions of the questionnaire in slightly different 
ways, to provide consistency checks* 

Description of Results 

On the question of establishing a clearinghouse to coordinatf^ in- 
formation about all MSG activities within the Department of Defense, 
respondents were generally quite favorably disposedj as shown in Fig* 
19* More than half thought it would be useful or highly useful. 

Twice we raised the issue of standardisation. The first timej it 
was in the question^ "What is your belief in the advisability of in- 
creasing DOD gaming and simulation activity for standardization?" Re- 
O spondents were asked to check one category on a five-point scale from 
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Table 6 

CROSS-TABULATION OF MILITARY SERVICE AND DIRECT COSTS 



Direcc Costs (Variable CSTDIR) 



Service 


■ $49K 


$50- 
$99K 


$100- 
S249K 


$250- 
S499K 


$500- 
$749K 


$750- 
$999K 


Sl- 
$2.49 
mllildn 


Unknown 
NR 


Total (%) 


USA 


12 


12 


4 


7 


5 


0 


1 


18 


59 (44.7) 


USAF 


5 


0 


2 


' 0 


0 


0 


1 


18 


26 (19.7) 


USN 


8 


10 


9 


1 


2 


2 


1 


3 


36 (27.3) 


DOD 


1 


1 


3 


1 


1 


1 


0 


3 


11 (8.3) 


Total 


26 


23 


is 


9 


8 


3 


3 


42 


132 


(%) 


(19.7) 


(17.4) 


(13,6) 


(6.8) 


(6.1) 


(2.3) 


(2.3) 


(31.8) 


(100,0) 



NQTEf Raw chi square * 47*42, with 21 deg freedom. Signlfieanca * 0.0008, 



9 

Table 7 

CROSi-TABULATION OF ^aLITAIY SERVICE AND TOTAL COSTS 



Total Costs (Variable TOTCST) 



Service 


< $49K 


$50- 
$99K 


$100- 
$249K 


$250-- 
$499K 


1500- 
$749K 


$750'- 
$999K 


Il- 
ia, 49 

million 


12.5- 
$5 
million 


Unknown 
NR 


Total (%) 


USA 


15 


6 


7 


1 


4 


1 


4 


1 


14 


59 (44.7) 


USAF 


10 


6 


2 


1 


0 


0 


1 


0 


6 


26 (19,7) 


USN 


4 


6 


10 


8 


1 


2 


2 


0 


3 


36 (27.3) 


Other 
DOD 


2 


1 


3 


1 


0 


2 


0 


0 


2 


11 (8.3) 


Total 


31 


19 


22 


17 


5 


5 


7 


1 


25 


132 


(%) 


(23,5) 


(14,4) 


(16.7) 


(12.9) 


(3.8) 


(3.8) 


(5,3) 


(0.8) 


(18*9) 


(100,0) 



NOTE: Raw chi square • 33,36, with 24 deg freedom. Significance ■ 0.0965. 
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F?g * 19— Respondent' opinions on MSG claarlnghou^^ 
(variable CLEAR) 

highly useful to highly harmful* The second tlme^ near the end of the 
questionnaire, the question was. phrased ^ "Is it premature to try to 
form a professional standards conmittee for models ^ games p and simu- 
lations? Is it needed? Would it probably do good or harm?" This 
time we asked for written commentary, which was later coded as highly 
useful, useful^ same as present ^ harmful, highly harmful ^ or no response. 
The comments gave insight into why there was such pronounced opposi-^ 
tlon to standardization I as evidenced in Fig, 20. 

We also asked for opinions on the initiation of regional centers 
to coordinate production^ operation, and use* Some 60 percent thought 
such centers would be harmful or highly harmful. Another question 
asked respondents whether the creation of external reviewing boards 
would be an improvement. Some 57 percent opposed the idea, terming it 
harmful or highly harmful and impossible to staff adequately. 

On each question we asked respondents to OTpllfy their categoric 
cal responses with written commentary. The prevailing attltuda of those 
opposed to clearinghouses was that it would add mi luinecessary layer to 
the existing bureaucracy* Roughly, the sentiment was* "In theory it 
sounds fine, but in practice it just won't fly»" We were struck by the 
extent of concern voiced by some of the more widely known and professional 
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Fig , 20— Respondents' opinions on standardization 
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of the raspondants about standardization of many kinds—of languages, 
data formats, or documentation. They considered attempts to standardiEe 
premature and feared they would stifle creativity. Incidentally, the few 
respondents who favored standardization were mainly users with little 
or only recent experience. The relatively negative reaction toward 
creating regional centers was also based on the concern that such cen^ 
ters would not only add bureaucratic problems but would also separate 
the work from those who know, need, and use it. Those who could see 
some merit in regional centers argued that they would save money on 
personnel and computer resources. Any benefit from coordinating data 
processing and computer software systems was mentioned only occasionally. 

One of the most interesting . sets of replies dealt with external 
reviewing procedures and boards. Some 17.4 percent indicated that they 
would be highly harmful* 39.4 percent suggested that they would be 
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harmful; and only one respandant chought tlia idea would be hlgHly use= 
ful* The argument against the review board was that it would be lin-- 
possible to assemble a iiigh-q uali ty panel to review specific models in 
sufficient depth to justify the time and effort. Many respondents feltj 
some In no uncertain terms ^ that they were perfectly capable of provid- 
ing all the review needed. Based on this survey-s results and informal, 
supplementary interviews, we doubt that they are right* 

Rejatjonship to Other Descriptors 

We tabulated all opinion questions against the respondent-s own 
rolu, to determine whetlif^r one-s location in the decision process liad 
any notlcaable effects ^ and against the service responsible for each 
MSO, to judge whether organizations had systematic preferences, 

While no significant relationships existed between respondent role 
and opinions about clearinghouses, external revieWj or technical coor^ 
dination, other matters were significanf. 

For exmnplej the correlation of respondent's role with opinions 
about standardization (Table 8) shows that f unders/sponsors most favor 
the creation of standards, closely followed by caretakers; designers 
and builders are the least favorable; and users are somewhere in be-- 
tween. The correlation of respondent-s role with opinions about creat- 
ing regicnal centers (Table 9) reveals that fundera /sponsors are again 
most in favors although less so than for standards. Again, designers 
and builders are least in favorj but they too are less strongly com- 
mitted than on the standardization issue. 

Tabulations of service opinions on the creation of clearinghouses 
are shown in Table 10; on standardizationi Table 11; and on technical 
coordination J Table 12* 

EVALUATION OF THE QUESTIONNAIRE BY RESPONDENTS AND RESEARCHERS 

We solicited respondents' opinions about the questionnaire, and 
in turn we rated the general overall quality of each respondent-s ra- 
piles. Because the questionnaire Is formidable, we checked to see 
what, if any, systematic relationships existed between the time re-* 
quired to fill it out, the respondent's appraisal of its and our 
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Table S 

CROSS-TABULATION OF RESPONDENT'S ROLE AND OPINIONS ON STANDARDIZATION 









Oplnl 


on (Variable STNKDl) 




Role 

(Variable RESROLl) 


Highly 
Uaeful 


Useful 


Same 


Harmful 


Hlehlv 
Harmful 


NR 


Total (%) 


Funder-Sponsor 


2 


10 


3 


3 


1 


0 


19 (14.4) 


User 


2 


10 


10 


17 


2 


1 


42 (31,8) 


Deilgnar^'Bulldar 


2 


k 


14 


14 


5 


2 


41 (31.1) 


Carataker 


3 


1 


2 


5 


1 


0 


18 (13.6) 


Contrpl 


0 


0 


0 


1 


1 


0 


2 (1.5) 


Other 


0 


1 


1 


1 


4 


3 


10 (7.6) 


Total 


9 


32 


30 


41 


14 


6 


132 


(%) 


(6.8) 


(2^».2) 


(22.7) 


(31.1) 


(10.6) 


(4.5) 


(100.0) 



NOTE: Raw ehi iquare ^ 56. 45, with 25 deg freedom* Significance ^ 0.0003, 



Table 9 

CROSS-TABULATION OF ^SPONDENT'S ROLE AND OPINIONS ON ^GIONAL CENTERS 



Opinian (Varlabla REGCEN) 



Role 

(Variable RESROLl) 


Highly 
Uieful 


Uieful 


Same 


Harmful 


Highly 
Hannf ul 


NR 


Total (^) 


Funder^Sponsor 


0 


5 


1 


8 


4 


1 


19 (14,4) 


User 


1 


4 


5 


21 


5 


6 


42 (31,8) 


Designer-Builder 


0 


6 


3 


18 


9 


5 


41 (31,1) 


Caretaker 


0 


2 


0 


12 


0 


4 


18 (13.6) 


Control 


0 


0 


1 


1 


0 


0 


2 (1.5) 


Other 


0 


0 


1 


1 


1 


7 


10 (7.6) 


Total 


1 


17 


11 


61 


19 


23 


132 




(0.8) 


(12.9) 


(8.3) 


(46.2) 


(14,4) 


(17,4) 


(100.0) 



NOTE; Raw chl square » 42.53» with 25 dig freedom. Signlfieance ^ 0.0157. 



Table 10 

CWSS'-MULATION OF MILITARY SERVICE OPINIONS ON CLEARINOTOUSES 





Opinion (Variable CLEAR) 


Service 
(Variable 
SERVIC) 


Highly 
Uieful 


Useful 


Same 


Harmful 


Highly 
Harmful 


NR 


Total (%) 


USA 


11 


25 


U 


6 


1 


2 


59 (44.7) 


USAF 


3 


8 


8 


1 


1 


5 


26 (19,7) 


USN 


3 


23 


2 


4 


4 


0 


36 (27,3) 


Other DOD 


2 


1 


7 


0 


1 


0 


11 (8,3) 


Total 


19 


57 


31 


11 


7 


7 


132 


i%) 


(14,4) 


(43,2) 


(23,5) 


(8,3) 


(5,3) 


(5,3) 


(100.0) 


NOTE J 


Raw chi 


Square * 


41,12, 


with 15 deg f reedoin 





Signifieance ^ 0,0003, 



Table 11 

CROiS-TABULAl .ON OF MILITARY SERVICE AND OPINIONS ON STANDAmiZATION 





Opinion (Variable STNRDl) 


Service 
(VarUble 
SERVIC) 


Highly 
Useful 


Useful 


Same 


Harmful 


Highly 
Harmful 


NR 


Total (%) 


USA, 


4 


19 


10 


20 


5 


1 


59 (44,7) 


USAF 


3 


3 


9 


4 


2 


5 


26 (19,7) 


USN 


0 


9 


7 


13 


7 


0 


36 (27,3) 


Other WD 


2 


1 


4 


4 


0 


0 


11 (8.3) 


Total 


9 


32 


30 


41 


14 


6 


132 


(%) 


(6,8) 


(24.2) 


(22,7) 


(31,1) 


(10,6) 


(4,5) 


(100,0) 


NOTEl 


Raw chi 


square ^ 


35.67, 


with 15 da| freedoffli 





^ Significance « 0,0020, 
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Tablf 12 



CROSS-TABULATION OF MILITARY SERVICE AND OPINIONS ON TECHNICAL COORDINATION 



Service 
(Vayiable 
SERVIC) 




Opinion (Variable TECH) 


Highly 
Unde- 
sirable 


Unde- 
sirable 


Indif- 
ferent 


Desirable 


Highly 
Desirable 


NR 


Total (%) 


USA 


9 


a 


7 


15 


19 


1 


59 (4^.7) 


USAF 


2 


3 


9 


2 


2 


8 


26 (19.7) 




5 


7 


6 


11 


6 


i 


36 (27.3) 


Other DOD 


1 


0 


6 


2 


2 


0 


11 (8.3) 


Total 

m 


17 
(12.9) 


18 
(13.6) 


28 
(21.2) 


30 
(22.7) 


29 
(22.0) 


10 
(7.6) 


132 

(100.0) 



Raw ehl iquare - 46.85, with 15 deg freedgm. Significance - 0.0001. 



evaluation of the quality of the replies. We also checked the two 
evaluation descriptors against the respondent's role and the military 
service responsible for the MSG. 

We assessed the quality of the responses before we knew how the 
respondent had evaluated the questionnaire. As shown in Table 13, 
overall distributions were cuite similar. Though the relationship is 
not scatistlcally significant, users and funders/sponsors were more 
favorably disposed to the questionnaire than were others; designers 
and builders were less pleased. We eKpected that to some extent, be- 
cause tiie questionnaire frankly favors issues of import to the user 
that many builders apparently ignore or hold in less esteem, ^ 

Our Qpinion of the quality of respondents* replies Indicates that 
users general] y provided us with better overall responses than other 
groups, «nd caretakers provided the poorest. 

Service'-specific opinions were not statistically significant on 
questions of regional centers, external review, or quality of the ques- 
tionnairt:. Our own evaluation of responses by serviae indicates that 
un Che average the Army did better than the others.' 15/59 or 24,4 per- 
cent were good or excellent. Perhaps that Is because of the demands 
placed on the Army aegmenc of the profession by the ad hoc Models Review 
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Table 13 

CROSS-TABULATION* OF RESEARCHERS' EV'ALUATION OF RESPONSES 
AND RESPONDENTS' EVALUATION OF THE qUESTIONNAIRE 



Ryieaf chef s ' 
Evaluat Ion 
(Variable IQC) 


Respondents ' Eval uat ion 


(Variable QUEVAL) 


Excellent 


Good 


Modal 


Poor 


Bad 


No Re- 
sponse 


Total (%) 


Excel lent 


0 


2 


1 


0 


0 


0 


3 (2,3) 


Good 


2 


7 


12 


3 


1 


1 


26 (iy.7) 


Modal 


1 


U 


10 


10 


6 


5 


43 (32,7) 


Poof 


0 


5 


5 


12 


8 


10 


40 (30.4) 


Bad 


0 




6 


7 


2 


3 


20 (15.2) 


Total 


3 


27 


. 34 


32 


17 


19 


132 


(%) 


(2.3) 


(20.7) 


(25.8) 


(24,3) 


(12,9) 


(14.4) 


(100. 0) 



NOTE: Raw chi iquare ^ 53,41, w-tth 20 deg freedom. Significance ^^O.OOOl, 



Committee, On the. other hand, other DOD and Air Force replies were 
poor or bad, 8/11 or 72,8 percent and 17/26 or 64,3 percent , respec-- 
tively. Navy responses were predominantly modal in qualityj 17/36 or 
47*2 percent. 

Time taken to complete the questionnaire was significantly related 
to both respondent and re.^earcher evaluations (see Table 14), Of the 
30 respDndents who though t the questionnaire was excellent or goodj 26 
or 86,6 percent required 15 hours or less to complete it. However, 
when researchers' evaluation is tabulated against completion time 
(Table 15) s the excellent and good responses, while concentrated under 
15 hours, are spread somewhat more along the time dimension* We con- 
jecture that if a respondent intimately knew his MSG, he could respond 
thoroughly and competently in under 15 hours; if not so knowledgeable 
but sympathetic to the purposes of the Inves tigationj he could pioduce 
high-^quall ty responses at some additional cost in time* It is also In^ 
teresting to see that as regards the three respondents who took over 
50 hours to complete the questionnaire , mutual enmity prevails! they 
rated the questionnaire low^ and, despite their protracted completdan 
Q time 5 we found their answers poor. 

ERIC 
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Table 14 

CROSS-TABULATION OF SESPONDENTS' COlffLETION Tim 
MD EVALUATION OF THl QUE3TI0NKA1R1 



Time 




Evaluation (Variable QUEVAL) 


v,variaDie ijl ^ 

(hr) 


Excellent 


Good 


Modal 


Poor 


Bad 


No Re- 
sponse 


Total (%) 


0-5 


2 


11 


9 


21 


8 


10 


61 (46.1) 


6-10 


0 


6 


10 


8 


4 


3 


31 (23,5) 


11-15 


0 


7 


6 


0 


0 


1 


14 (10.6) 


16-20 


1 


2 


3 


0 


■ 1 


0 


7 (5.3) 


21-25 


0 


0 


3 


0 


0 


1 


4 (3.0) 


26-30 


0 


0 


1 


1 


0 


1 


3 (2.3) 


31-35 


0 


0 


0 


1 ■ 


2 


0 


3 (2.3) 


36-40 


0 


0 


1 


1 


0 


0 


2 (1.6) 


41-45 


0 


0 


0 


0 


0 


0 


0 (0,0) 


46-50 


0 


1 


0 


0 


0 


0 


1 (0.8) 


a 51 


0 


0 


0 


0 


2 


1 


3 (2.3) 


NR 


0 


0 


, 1 


0 


0 


2 


3 (2.3) 


Total 


3 


27 


34 


32 


17 


19 


132 


(%) 


(2.3) 


(20.7) 


(25.8) 


(24.3) 


(12.9) 


(14.4) 


(100.0) 



TEi Raw chl iquare ^ 82.64| with 55 deg freedom. Significance ^ 0.0125. 
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Table 15. 

CROSS-TAIULATION OF QUESTIONNAIRE COMPLETION TIKE 
■ AND RfiSlARCHERS' EVALUATION OF RESPONSES 



Time 
(Variable QT) 
(hr) 




Evaluation 


(Variable IQC) 




Excellant 


Good 


Modal 


Poor 


Bad 


Total (%) 


0-5 


0 


7 


20 


22 


12 


61 (46.1) 


6-10 


2 


5 


15 


6 


3 


31 (23.5) 


11-15 


1 


7 


3 


2 


1 


14 (10.6) 


16-20 


0 


0 


4 


1 


2 


7 (5.3) 


21-25 


0 


3 


0 


1 


0 


4 (3.0) 


26-30 


0 


1 


0 


2 


0 


3 (2.3) 


31-35 


0 


1 


0 


2 


0 


3 (2.3) 


36-40 


0 


0 


1 


1 


0 


2 (1.6) 


41-45 


0 


0 


0 


0 


0 


0 (0.0) 


46-50 


0 


1 


0 


0 


0 


1 (0.8) 


S 51 


0 


0 


0 


2 


1 


3 (2.3) 


NR 


0 


1 


0 


1 


1 


3 (2.3) 


Total 


3 


26 


43 


40 


20 


132 


(%) 


(2.3) 


(19.7) 


(32.7) 


(30.4) 


(15.2) 


(100.0) 



NOTEi Raw chi iquare ^ 61.03, with 44 deg frsadcra, 
Signlfleanee ^ 0,0512. 



III, CQNCLUSigNS 

The general discipline known as gaming/simulation is highly diverse. 
It encompasses at least four different subjects that have different 
criteria for validation and different measures for cost and effective- 
ness. That makes It a difficult entity to grasp analytically* Never^ 
thelesSj in the following pages , we venture our opinions on the slg-- 
nlflcance of the regults of our survey. They are sumtnariged under the 
same lubitantive headings as were the results analysed in Sec* II, 

mG PURPOSES 

Weapon system evaluations , mostly built by the individual military 
services for their own exclusive purposes and usej predomlnatej both 
in absolute level of activity and in total expenditures. Ironically s 
expenditures and public knowledge about gaming and simulation, are in- 
versely related* Free-form gaming has received great and continuing 
notoriety^ but Its expenditures are trifling compared with expenditures 
on all^machine, technical evaluations* Many models, simulations, and 
games are literally unknown outside a small user^producer coterie. 

The degree of knowledge that anyone of even this in- group may have 
about a particular MSG is evidently limited. Many poor^ incomplete^ 
and low--eonf idence replies were obtained from responsible and knowledge- 
able professionals. That several respondents took upwards of 70 hours 
to complete the questionnaire reinforces this view. 

Also worrliome Is the amount of what is essentially research money 
that is being spent in the absence of rigorous and accepted research 
standardSi^ As a result j basic knowledge about both substantive and pro- 
cedural matters ±u neglected* Very little is being spent on validation 
efforts and on basic research about MSG methods i dataj and useSp with- 
out which the credence of gaming and simulation suffers. 

In our opinion^ the copious in-house production of technlcal-^eval^ 
uation models Is being pursued without sufficient attention to quality 
and scientific rigors 



o In many cases, in--house work has ignored Eclentific standards 

of data collection^ management ^ and validation. 
0 Documentation of in-house work is often very poor;- this fail-- 

ing is frequently rationalized on the grounds that the work is 

not meant to get outside the builder's shop, 
o About half the sampled MSGs were not externally reviewed; most 

respondents reject the need for such reviews on bureaucratic 

rather than scientific grounds. 

PRODUCT IQN 

The evident preference for large, all-machine modela and aimula^ 

tions is questionable on several grounds. Large-scale , finely detailed 

MSGS that try to deal with problems having significaai: uncertainties 

* 

may only serve to generate errors, not clarifying anythingp Given 
what appear to be weak-to^poor data, extremely fine temporal and spatial 
levels of model resolution, and low levels of demonstrated concern for 
supporting research s the MSGs produced may have doubtful reliability. 

Large modelo are usually complicated ^ expensive to build and use, 
take extended periods to operate and Interpret , and are the least sci- 
entifically defensible* They quickly begin to suffer from the disor- 
ganization created by changes in purpose and personnel, bad documenta= 
tion, gaps in logic, and problems of data-base preparation ^ maintenanca, 
and validation. 

If large models must be produced, the key to control seems to be 
in continuity of personnel. Changes of personnel have significant ef- 
fects, ITsage decreases because no one- knows what a model is supposed 
to do, how it does It^ or why. Where sunk costs are great, there Is a . 
tendency to use a large and eKpensive model anyway^ even though none 
of its caretakers can determine its validity for new applications* Doc- 
umentation should ameliorate this problemi but it seldom does. 

For a technical discussion see John W, Tukey, "The Propagation of 
Errors, Fluctuations and Tolerances i Basic Generalized Formulas," Re- 
port No, 10, Department of Mathematics, Princeton University; for a 
more practical view, see William Alonso, *'The Quality of Data and che 
Choice and Design of Predictive Models," in G, C. Hemmens, ed*, Urbmi 
DeVBtapmBnt Modgl& ^ Special Report No. 197, Highway Research Board, 
Washington, D.C., 1968, pp, 178-192, 



OPERATIONS 

We believe that documentation Is considerably worae than the ques-- 
tldnnaire raplias indicate, Documantation standards, requiring that 
dDcumentatlon meet certain specific criteria, cost little axtra for thatr 
large contribution to battar management control. The need is particu-' 
larly great in the casa of very large ilmulations. When generations of 
programmers perpatuate Inconsistencies and errors in sloppily updatad 
documantation, all can be lost. 

>tany capabilities built into these devices have not been subjected 
to validation* Not only is their empirical basis dubious or admitted 
to be lacking p but few efforts are being made to collect missing or 
questionable input data or to execute sengltlvlty analyses according to 
an appropriate axperimental design. The lack of sensitivity analysis 
is related to deficiencies in estimating the validity of input param- 
eters* Neither of these matters seems to be taken ierlously. There 
is a less than 50/50 chance that a sensitivity test will be done, and 
*^en It is done J there is frequently no record of the outcoma* 

Most of Che surveyed MSGs have not been lubjected to any external 
review, with the result that many contain implicit and intanglbla ln=- 
puts ^ose existence and rationale have not been documented In any way. 
The io-called "institutional mmory" In the general system Is not very 
well developed. Even about MSGs of fairly recent vintage, respondents 
seem unable to answer technical and cost questions very well or with 
much confidence. 

One colleague has suggested only half facetiously that validation 
Is a happy customer. If so, is ultimate validation a follnw-on con- 
tract? A comprehensive review of data^validlty problras is needed, and 
iome strong corrective measures should be instituted. Advocacy rather 
than scientific preferences seems to predominate. 

4 

USE 

In our opinion J the level of professional communication is danger- 
ously low, There Is a great need for better coordination, documentation, 

J. Mayberry, "Principles for Asgessment of Simulation Model Va- 
lldlty," in A, I, Siegel, edp, Proa&edingB of the Symposim on Comput&r 
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and studies of use at the operational, experimental, and administrative 
interfaces. It is not enough that a study be finished according to for-- 
mal contract specifications; what becomes of the study and how it is 
used are far more important considerations. Learning, both kinds and 
amounts, must be evaluated so that resources can be expanded more ra- 
tionally. Perhaps it would be worthwhile to declare a moratorium on 
expenditures for new studies imtil existing ones have been properly 
evaluated. Technical weapon evaluations that are either unused, little 
used, or miiused, for exOTple, may be worse than no studies at all. Im- 
prudent management policies (for Instance, the high regular turnover of 
key personnel in some military activities), coupled with uneven documen- 
tation standards and procedures, may account for the ineffective use 
of models. If no one remembers why a particular model was built, for 
whom, and how it operates. It may be used incorrectly, or, worse, peo^ 
pie may start from scratch to build^ a model that will do the job it 
was in fact designed to do. If there is little or no documentation, 
the potential waste becomes a practical certainty . 

It is difficult to determine what influence these MSGs have had. 
As noted, half of them did not ^ even result in a direct briefing. Writ- 
ten comments roughly suggest that they "did the Job"; however, little 
information was given on the importance of the "job" and the policy 
decisions that depended upon it. Documentation accompanying any MSG 
should indicate what it has been used for, who has used it, when it was 
used, and so forth. 

. The dearth of written scenarlog and explicit consideration of in- 
tangible and uncertain elements is of concern. It is al.1 too easy to 
misuse well-specified numerical modals by changing the problem context 
or Interpretation. A model that is impressively "good" in one contex*: 
may be inappropriate in another context, even if specific data for the 
hard numbers are adequate. That depends upon the skills and inter- 
face of those who know the original uses of the model, those who de= 
termine the new purposes and applications, those who set the soft num- 
bers, and those who provide the background interpretations. There 



ERIC 



Simulation as Related to Manpower and Pereonnel PZanning, Naval Per- 
sonnel R&D Laboratory, Washiniton, D.G., July 1971, pp. 157-165. 
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is lictle evidence on how those taskg are being done, if at all. It ap- 
pears that the engineering, applied-science bias of most of the builders 
and Caretakers we sur-veyed would make . replication of their MSGs risky. 

What all of this means can be stated briefly: there is no substi- 
Lute for people who know their business. One of the real dangers that 
we perceive is the poverty of the scientific interface between those 
who know their business and those who are trying to generate business. 

The Department of Defense has nurtured a group of specialists who, 
having made their share of mistakes^ are doing some competent and use- 
rul work. But there are few communication means by which their knowl= 
tHige can be shared more broadly with those in the civil sector. 

Notwithstanding this emergent professionalism, the tenuousness of 
iTiUch of the data being used, the immature extend and level of vallda- 
Lion, and the relative neglect of such ImporCant scientific and opera-- 
tionnl procedures as sensitivity analysis and scrutiny of the appropri^ 
' .^reness of work for specific operational environments and scenarios, 
make If: easy to infer that advocacy rather than scientific preferences 
prevails, Can scientific content be improved and unfounded advocacy be 
reduced? Generally, the answer is yeSj although the problem is dlffi- 
culi:. 

One deairable innovation would be to make the advocacy process two- 
sided rather than one-sided as It is now. For example ^ the quality 
T'f disccurse on weapons procurement would be improved if congressional 
parLlcipants as well -as DOD -proponents had consultant advice and pro- 
fessional assiscance. In other words, improve existing methodi and 
procedures for challenging the assumptions and quality of work done in 
support of any one position, system, or decision* Lawyers and accoun= 
tants may be able to win debating points from the englneersi generals, 
and mathematicians, but that suggests a far from rational allocation 
process J much less an optimal one. 

The recent discussions between George Rathjens and Albert 
Wohlstetter on the proposed Safeguard ABM system before the [J,S, Senate 
illustrate the benefits of having both sides on a question technically 
compecent and informed with comparable data, The entire issue on Op^ 
^ratLr^ya R&SBiWah, Vol, 19, September 1971 , is devoted to their intelli-' 
gent debate. 

EKLC 
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COSTS 

Casting is deficient. Granted^ cost accounting is difficult to 
do in a process as complex as the initiationj productionj operation, 
and evaluation of ^^SGs; nevt^rtheless , for better management control^ 
some record of cost must be kept on each MSG, It need not be compli- 
cated; even crude figures within ±100 percent of the actual would 
be an improvement over current information. 

More than one-third of the respondents j over a range of diverse 
questionsj did not or could not consistently reply to the most alemen-- 
tary cost questions. Those that did respond ^ moreover , had low con- 
fidence in their answers, 

Wliose responsibility and in whose interest is cost control? De- 
velopers have little reason to be interested in costs except as thev 
contribute to their own revenues. Questions about alternative methods 
to modeling and simulation are seldoni explored in the operational set- 
ting. Current procedures do not seem to include formal consideration 
of whether there is a cheaper^ easier way to proceed or whether model- 
building is really the most appropriate technique. The fiL.,c shou3.d be 
a managerial concern* the second should be dealt with at the technical 
level , 

That responsible persons were unable to .supply much rudimentary 
cost data suggests that cost accounting has been neglected. It is 
evident that cost data^ perhaps conceived of primarily as relating to 
investment J quickly become separated from the work itself. What re- 
sults is that users s caretakers, and even builders have but vague no^ 
tions of cost a year or so after the work has been undertaken. 

Curiously, there appears to be a magic (i.e^^ relatively easily 
fundable) budget amount between $200 » 000 and $300,000. Judging from 
our survey data, it would be better to apply for two separate budgets 
for two different models at $200,000-$30O,00Q each than to apply for a 
budget for one model at $400 , 000-^$600 ,000 , Families of MSGs have been 
maintained for years with separately labeled components funded sepa- 
rately and used and evaluated independently* Actually , many arc merely 
increments of the same basic work and should be considered as sucih. 



It Is interesting that most money is spant where professional 
visibility and active participation by high-echelon personnel are mlni= 
mal. That raises an important question about the effectiveness of ex-= 
penditures. Are we trying to compare incommensurables? 

Machine models and gimulations appear to favor so-called value^ 
free engineering work and are produced by bright lieutenants, captains, 
majorsi or possibly colonels in cooperation with civilian contractors. 
Such work can easily lead to larger studies that became- briefings for 
colonels and generals and their civilian counterparts. Information 
about the context, such as verbal descriptions and scenarios, and de^ 
tails about purposes and limitations of a model are seldom spelled out, 
Man^machine exercises, in contrast, are frequently used for teaching or 
training in the staff colleges. As a result, there is some chance that 
a two- or thrae^star general or admiral might remember or have learned 
aomething as a result of a man=machlne exercise he participated in. 
Such activities may also be used In an eKperlmental laboratory where 
the personnel employed are not necessarily military and where the pur= 
^poae is altogether different. As noted, the least expensive activities 
are the political-military exercises that at one time or another may 
have commanded the attention of the highest-level personnel. Whether 
free-form military games accomplish anything besidei entertainment is 
an open question that deserves further study. 

Little attention has been paid to what is meant by claims that a^ 
certain study has influenced policy. An imaginative briefing by the 
likes of Herman Kahn or a political-military exercise run by an Albert 
Wohls tetter with high-level participants may have had more influence 
on policy than most multl-mllllon-dollar models and ilmulatloni. Then 
again, neither may make any difference. Much depends on the timeliness 
of the exercise and its relevance to current problras. 

questions like the foregoing must be made eKpllclt. Then It may 
be possible to take a more objective look at the routine expenditure 
of millions of dollars for middle-level, engineering- type MSGs. 
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ly. RECOWIENDATIONS 

From our analysis of the survey results, we have identified the * 
fDllowing issues as being critical to resolve if the standards of 
gaming and simulation are to be raised. 

ADVQCACY VERSUS SCIENTIFIC VALIDATION 

The process of building and using MSGi provides an important illus- 
tration of model-^bullders ' neglect of science in the interest of advocacy. 

As long as model-builders do not question the environment set for them 

\ - _ ._ 

by those soliciting the work, practically any point of view can be sup- 
ported by selecting appropriate ''guesstimates'' about the environment. 
Accordingly, there Is a need for open^ regular, and more rigorous review 
of the models, games ^ and simulations that are being built and used. 
Procedures should challenge the validity of data inputs that are now so 
routinely and unquestloningly used. 

Communication between civilian and military segments of the pro- 
fession must be improved, A profeisional advisory group at the level 
of Congress or the GAO appears to be desirable to enhance the dialogue 
between the Congress and DOD, 

COST ACCOUNTIMG 

Given the little or no ins titutlonal memory about the cost of build- 
ing, running, updating^ and evaluating MSGs, any effort would prbbably 
be an improvement over the present poor-to-nonexlstent procedures of 
cost accounting. The accounting definitions and categories we were 
forced to create for this survey may be useful as a point of departure. 
At the leasts a brief cost dossier should be kept with an MSG throughout 
its eKlstence. 

EXTERNAL REVIEW AND PROFESSIONAL STAITOARDS 

Professional reviewing is a critical and much-neglected means of 
quality control. To those who claim that it merely adds an extraneous 
bureaucratic Impediment to ^'getting the job done," we cite the efficient 
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''and competent efforts of the Army Models Review Committee as evidence 
Co the contrary, Repettably, that committee-s work was ad hoc and 
temporary, when persist ence is called for. 

The Joint Chlafa of Staff might well be able to assume an MSG re- 
view function for all the services, VThat Is required are qualified pro- 
fessionals to serve am a long-term, continuing basis and strong incen-^ 
elves to insure adherence to scientific standards of evaluation. Such 
a group should also d@al with questions of standards ^ such as ^ When is 
sensitivity analysis needed? and What constitutes data validation? 

DOCUMENTATION 

Documentation is. largely of uneven quality , not available , or non^ 
existent. At a minlmwn, an MSG-s documentation should include the 
following: the progrmi listing; flow charts; variable listings, defi- 
nitions , and sources I the program deck with comments and caveats about 
operating quirks and special library or input/output routines; the op- 
.era^Q3c's manual; the programmer ■ s manupl- the player ^s manual if the 
MSG is man-machine oc free-form; the pertinent analysis roucines used 
to reduce data generrted by the MSG and to estimate input parameter 
values for the ^ffiG; appropriate data reflecting what the MSG cost to 
construct J updatej and run; and the register of critical personnel in-- 
volved in MSG tnitiaWo?i~who wanted it built, for what reasons; pw^ ^ 
duation—thB identities of the master modeler and the model team and 
%AiBt validation procedures they used; operations — the history of pro- 
fessional review by persons external to the builder-user; and vMe—yjho 
used it, when, and with what purpose and outcome. 

Many will complartn that this list is too long, is burdensome, and 
would impede the worlt at hand. Nevertheless, conscientious and con^ 
sistant attempts to aemplete it will improve eKisting management prac** 

tices and nay in time open the way to improvements in the state of the 

art and the state of knowledge* 



Army Models Review Committee j liavieW of Seleated Aimy Mode la ^ De-^ 
partmc'.it of the Army^ Washington, D,C., May 1971, Chapter 8, 
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REDUNDANCY A ND STANDARDIZATION 

we find no evidence that redundancy is exaBBsive, Communication 
amoung MSG builders is poor and needs improvemett; howevers even if it 
were good ^ a certain amaunt of redundancy would Tbe desirabiej and that 
amount does not appear to have been consistently .exceeded, 

S taadardization should not be confused wi^ adherence to profes^ 
sional standards. We recommer.d atrongly against the firsts as it is 
premature, We urgej however , that efforts be irofle to promote the second 

A CENTRAL CLEARINGHOUSE? 

We recommend agains t the creation of an adffitlonal bureaucratic 
operation for the clearing of work on MSGs* HoiEverj thers is a le- 
gitimate need for a professional focal pointy wkicii could be provided 
by the type of JCS professional review board rMBDnuiiended above. 

Work in gaming and simulation is fragmentai to an unnecessary and 
unhealthy extent. We urge the joint usage of bffiGs. By that we mean 
the sharing of MSG construction and use among flie military services and 
government agencies; the scientific replication tof MSGs| collective at^ 
tempts to account for and reduce costs; and the ^sharing of special over- 
head charges for large-scale, general-purpose mmputational ays terns , 
even more than is now done via the NMSCCS deviM.. Joint usage promises 
the more efficient use of scarce professional telent and the diffusion 
of professional standards. The creation of a KSS Models Review Coiranit- 
tee based nn the Army prototype would be a step in the right direction. 

RE.SEARCH 

Basic research and knowledge is lacking, TOie majority of the MSGs 
sampled are living off a very slender intelleataial Investment In funda'^ 
mental knowledge* While this is probably not ifiie time to expect much 
funding for basic research j tha need is great fer work on topics such 
as simulation methods s data validation, sensltlwlty analysis, and sta- 
tistical tests for simulation outputs* As re^icds analysis, study is 
needed on formal models of combat ^ such as altocation and search game 
theory. In the -'softer'' subjects that bear dlOTBCtly on applied MSGSj 
there is a need for studies of panic behavior ((tthe "breaking point" 
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hypothesiSi for eKample) ^ threat and confrontation, and eipecially hu- 
man factors and motivation* At another level, work is required on 
basic questions of usej both for particular MSGs and for whole families 
and classes of MSGSi 

The "image" of research needs refurbishment among funders and 
builders in the professional community. Research appears to be so stig^ 
matized that one can scarcely acknowledge sponsorship of a pure research 
project without bracing for criticism. 

The need for basic research is so critical that If no other fund- 
ing were available we would favor a plan to reduce by a significant 
proportion all current expenditures for MSGs and to use the saving for 
bailc research. 

mG SIZE 

There is every indication that the larger MSGs have been of little 
utility. The aizej length of time under development , and generality of 
an MSG all appear to be directly related to the difficulty of control- 
ling, validatlngi and using it. Undesirable outcomes resulting frbm 
changes in personnel , bad documentation ^ poor conceptualiEation ^ and 
poor professional communication and review are only exaggerated with 
large MSGs. We recommend that standards for approving the construction 
of large-scale MSGs be much more stringent than for smaller projects. 

We believe that large-scale MSGs tend to lack the capability of 
handling icenarloi and other hard--to-quantlfy elements. Funds would be 
better spent on the basic research to acquire that capability than on 
the premature constuctlon of large programs. 

FREE-FORM MP MAN-I^CHINE G^ING 

We believe that it is time for an assessment (preferably by mill-' 
tary and nonmilltary' personnel ^ "hard" and "soft" aclentlsts) of the 
role of free*^form and man-machine gaming in DOD work on MSGs, relative 
to that of all^raachine simulation t We suspect that £ree-=forni gaming 
deserves more prominence, 

Man-^machlne g^ing for operational purposes appears to be rela= • 
tlvely expensive and not heavily used* For training and teaching, 
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both man-machine and free-forin gaming appear to have been relatively 
undersupported. 

Me recommend that an investigation be iponsored to expiore the re- 
lationship among multiple uses of MSGs. Specif Icaxiy, especially given 
the projected size of the new Naval War College gimlug facility, the 
relationship among training, evaluation, operational, and research uses 
must be better understood. 

GAMING IN THE CIVILIAN SECTOR 

In spite of some negative findings, we see distinct growth in the 
state of the art and the emergence of professional standards. We are 
troubled, however, by the lack of professional communication between 
and outside the specialized, in-house model-building shops. It seems 
likely that many of the mistakes committed in the application of sim- 
ulation and gaming to DOD problems are about to ba re-committed in 
civil-sector applications. Given the lack of communication that sug- 
gests, it may be fortunate that funding for MSGs from agencies such as 
HEW and HUD is small relative, to that front DOD. 



* 

Indeed two pioneering attempts by HUD have been appraised and 
found wanting in many of the same ways as the MSGs we report on here. 
The problems are apparently common. See Garry D. Brewer, The Politic 
cian,^ the Bumauovat, and the Cone ultmt: A Critique of Urbmi Problem- 
Solving, New York: Basic Books, forthcoming In 1973. 
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AppendiK A 

QUESTIONNAIRE: MODELS, COMPUTER MA CHINE, SIMULATIONS , 
GAMES, AND STUDIES'" 



PREFACE 

The purpose of this questionnaire is to aid the interested profes- 
sional in describing, cliaracttrizing, and analyEing his game, model, or 
simulation. It is a first andj hopefully, useful step in the clarifica- 
tion of professional standards in the work on gaming and simulaticn. 
Furthermore, the questiDanaire is designed so that it might also serve 
as a device for communicating and cataloging different games and Simula-^ 
^ tions in a format that encourages easy Interchange of information. 

The questionnaire format has been adopted for three purposes. 
(1) In the course of our ongoing investigations we expect to use this 
document as a questionn-iire. (2) Stress in design has been to produce 
a categorizatipn scheme for the description and classification of games 
and simulation in general. The goals are to help establish professional 
standards and to explore the possibilities of deveioping a reasonably 
ggod classification and consistent description that covers many games of 
different varieties. (3) It is our belief that che compiling of a large, 
consistent sampla of many games and simulations for the purposes of anal- 
ysis, evaluation, information interchange, and the construction of pro- 
fessional standards is overdue. The work Involved in doing so is both 
large and onerous. The handling of large quantities of data calls for 
at least partially computerized procedures. The format we are pre- 
senting here v;as designed with this type of data processing in mind. 

This to t?ie aurvey imtrwnent^ p^^roduaed in its mtirety. (It 
wciB ppeviously published^ by the sm& authops iwdBr the fcvm title ^ as 
Pr4872^ The Ear^ Cor^pomtionj July 1972 J Page nwnbers have been 
changed to bs-in sequenas idih this mport. Respmses to many of the 
questiono in Parts I and II are indiQated in italics. 



This questionnaire had as an original purpose the description and 
classification of games in general. Because the preponderance of all 
availabla resources for modeli, siraulations^ and games is spent by the 
Defense Department and its various derivates, we have redesigned and 
reorganized the instrmnent to reflect these activities better". Some 
questions ware reworded, some were addedj others were discardedi At 
this point in the process, separate questionnaires for computer, all- 
machine models, and for man-machine and iMnual games were constructed. 

These modified questionnaires were applied to ten representative 
DOD models^ simulationi, and gameSi Several Rand Corporation games 
were also sampled. This "shakedown" prompted another round of *',valu= 
ation and redesign. The present version of a single questionnaire is 
the result* Although the emphasis on DOD is evldents we believe that 
a hard core of generally applicable information remains. 

Questionnaire conceptualization and design is a potentially diffi- 
cult undartakingi This particular questionnaire covers a compleK, diverse 
changing^ and specialised body of Imowledge and practice; consequently , 
the difficultiei found in its construction have been formidable. For 
instance, even at the most primitive daflnitional level there is little 
but cloying disagreement. What is a model? What difference is there 
between a model and a simulat' -rn? What is a game and when is it not a 
model, and vice versa? Indeed the. semantic game presently appears to 
take precedence and to substitute for the real game all too frequently. 
Other indicators of an unsettled but emerging prof esslonallsm abound. 
What this means is that construction of a questionnaire such as this is 
hard work; and no matter what results, it will have shortcomings, prob- 
leras, and probably more than a few errors* We aclmowledge the wealmesses 
and welcome comments, suggestions, and corrections from others concerned 
with gaming and simulation* 

Parts I, Hi and IV are designed for all types of models 3 games, and 
simulations. Manual and man-machine activitie^4 necessitated the addl'- 
tional questions In Part III as well. Further work is in progress In the 
categorization of questions more specific^ly aimed at the uses of gaming 
and simulation for teaching and for experimental purposes. We recognize 
that this questionnaire cannot be regarded as adequate in providing a 
means cf analysis if the two major uses of a game or simulation are teach-- 
ing or research alone* ^ 
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PURPOSES OF THE QUESTIONNAIRE 



1, To give an overall quick picture of the purposes ^ usa^ benefits, 
and costs of DOD activities in gaming, s^iiulEtioni and the use of 
models, 

2i To provide an initial description and classification scheme for 
a professional catalog for gatnaSi siTnulatlonSi and niodels for use 
in: 

0 Quick professional interchange of information on 
the characteristics of models* 

0 Aid in construction and estitnation of model 
characteristics for new model-builders. 

o Aid in the evaluation of previous and current 
activities. 
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INTRODUCTION 1 TERMINOLOCT 



The state of the profession is such chat no clear 
agreement on fundamental terms exists in all ol: 
Che various activities using models^ simulations, 
or games* Acknowledging this facCj we advance 
the following definitions to provide guides Co 
/ indicate the type of work this questionnaire covers. 

Coaming: A gaming exercise employs human beings acting as ttieni- 
selves or playing simulated roles in an environment 
^ which is either actual or simulated. The players may 
be experimental subjects or participants in an exercise 
being run for teaching, operational, training, planning, 
or other purposes. This questionnaire is designed to 
describe games constructed for these various purposes. 

War Gaming I One of the major applications of simulatlnn is war gaming. 

A war game is defined by the Department of Defense as a 
simulation of a military operation .nvolving two or more 
opposing forces and using rules, data, and procedures 
^ designed to depict an actual or assumed real-life situa^ 
tion. It is primarily a technique used to study problems 
of nillitary planning organisation, tactics, and strategy, 

A war game can be conducted to cover the entire spectrum 
of war, i.e., politico^military crises, general war, 
or limited war. Tht game may be based on hypothetical 
situations, real-^world crises, or current operational 
plfms* Some games are designed for joint operations by 
two or more military services, some are for uge by a single 
service, and others may be used by individual Army field 
cornmanders or even by division or battalion conmianders* 
The level of coffimand at which the game is to be played, 
of course, influences the type of units to be represented 
and the scope of operations to be conducted. 

There are three types of war gaMs in commqn use today i 
the training game, the operational game, and the research 
game. The training game is the least complex and Is de^ 
signed to pfovlde the participants with decisionmaking 
opportmiities similar to= those that may be experienced 
in combat. The operational game deals with current 
organizations, equipment, and tactics. It is mora complex 
than tne training game, uses inputs that are based on 
known quantities, and is used to test operational plans. 
The research game, which is the most complex of the 
three types of games, requires careful preparation to achieve 
maxima objectivity and usually is designed to study tactical 
or strategic problems in a future time frame, 



A war game can be accomplished manually, can be computers- 
assisted, or can ba wholly computerized. Manual games 
are played using symbols, pins, or plecas to represent 
forces, weapons, and targets on maps, maphoards, and 
terrain models. A compuLer^asslstod game is a manual 
game using computerized models, which fre^ the conLrol 
group from manv repetitive anu time-consuming computations. 

Computer games are based on predeterTnined procedures . 
All simulation of conflict is done by the computer tn 
accordance wi th the de tai 1 ed ins t ructions con tained In 
the computer program. Thy primary advantage of computer 
gaming is that the same gituation can be simulated manv 
times under differing conditions, to observe varlabnit%* 
of results* A computer war game reqtiires rlie use of a 
war game model (i.e.j computer program), which contains 
all the rules ^ procedures, and logic required to conduct 
the game . 

Simulation: Simulation involves the representation of a system or 

organism by another system or models which is dGsigned to 
have a relevant behavioral similarity with the original 
system. Gnmes utilize a simulated envirenment or simu^ 
lated roles for the players or both. In general, all games 
are simuli^tlons. However,' it is not particularly useful 
to use tho reverse categorization. In other words, not 
all simulations are usefully regarded as games. Computer 
simulations stimulating conflict or cooperation (su'^h as 
completely computerized battle models) are usefully con- 
sidered as games. Possibly, so are soma logistic or resource 
allocation models where the single (auLomated or live player) 
team may be regarded as struggling agaitist a statistical or 
strategic opponent called '^Nature." The borderline is ^not hard 
and fast; however , we would expect to Itave out a straight 
industrial production scheduling machine simulation from our 
category, for example. 

Computer simulation is an analytical technique which 
Involves the use of mathematical and logical models to 
represent the study and behavior of r^al-world or hypo= 
thetlcal events, processes, or systemt^ over extended 
periods of time. 

Simulation provides the means for gaining experience and ' 
for making and correcting errors without incurring the 
costs or risks .of actual applicaticm. It offers opportu= 
nities to tes t theories andv propos«id iiiodif ications in 
systems or proceases; to study organizations and structures; 
to probe past, present, and futuru events; and to utilize 
forces that are difficult or impracticable to mobilize, 
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Simulation tharefore is of value both as an educational 
device and as a means of discovering improved methods* 

Simulation should be used when (1) it is either impos- 
sible or extremely costly to observe certain proceisei in 
the real worldj (2) the obsarved syatam Is too complex 
to describad by a set of mathematical equations, 

(3) no itraightfo ward analytical techniqua exists for 
solution of appropriate mathematical equations, and 

(4) It Ig either impoisible or very costly to obtain 

data fo^ the more complicated mathematical models describing 
a system* 

On the other hand, sinulation should not be used when 
(1) simpler techniques exlit, (2) data are inadequate , 
(3) objectives are not clears (4) there are short-term 
deadlines J or (5) the problems are minor. 

Contract Studies 
6 Analyi^ei* The words "studies and analyses i" as used in this 
questionnaire, refer to those studies and analyses 
done by contract or by grant and which deal with the 
systematic and critical examinations of various subjects i 
Scudiea and analyses often require advanced analyttcal 
techniques to integrate a vfiritty of factors and to 
evaluate dat«. Their purpose Is to provide greater 
unders tandirig of altemative organizations, taccics , 
doctrines I policies, strategies, proceduras, systems, 
and programs , 

Models Conducting a CDmputer or CQnTputer^assiated war game or 
related simul^ttion requires the use of a computerized 
model. As us^d in thii conteKt, a model Is a document 
or program cpntainin| all rules, methodology, techniques, 
procedures i and lo|ic required to simulate or approximate 
reality, A computeriied model is a computer program or 
series of programs, dailgned to slmulace the logic of 
actionB or interactions of an environment or a context 
and prortde the reiults to player personnel for aubsequent 
analysis, 

MSG^ Tc avoid the ambtguitles of ^'local*' daflnitlonal usagt 
pecaliarlties s we have elected to use the shorthand 
virston "MJG" to stand for mQdel/simulation^ear. in the' 
femainder. of the questionnaire* This, wa hops, will 
, facilitate present communications In lighc of ihe still 
unresolved I serioui definitional problerai* The respondent 
is encouraged to substitute for himself whatever local 
tenn stems appropriate when he encounters the "^ISC" label. 
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INSTRUCTIONS 

In filling in this questionnuire , as many questions as posslhk? 
have been designed to be answered by marking one or mofe relevanc 
categories In each quescion (e.g., circlinR or nmrklng a numbL.r). 
This should help to increase the speed wiLh which the questions can 
be answered. Furthermora, it is easier to use computer nssigtanct- to 
process questions of this type. However, in many instances the wurda 
used for the categories do not quite reflect the properties of thu 
game, sitnuiation, or study being characterized. 

The respondents are requeBted to mnrk the most relevant cacegories, 
then to make any written comnientary modifying the answer or suggesting 
an improvement of the question. 

Notes are provided on pages opposite the qusstions. 

Use one questionnaire for each man-machine game or simulation, 
machine (or analogue) simulation, or study, Parts I, II, and IV of 
this questionnaire are designed to be answered for aH MSGs". Part 
III is added to accomraodate the particular characteristics of man- 
machine and manual games. 

Note nil Please note the amount of time taken to fill in eaoh 

questionnaire and return to Question iH to record this information 

upon completion of a questionnaire. 

Note iiZt To the right of each question there is a three-intorval 
confidence scale. If you are certain of your answer mark an "X" 
at the extreme right on the scale. Please mark your confidence 
level for all questions where the scale is indicated. 
Low Middle High 
L I 1 



J. 



Confidence Level 



On a scale of 0-1 your answers will b.. ^;r»i;-.ted as follows: 

Virtually certain .9 - i 

High , ."' _ .9 

Middle . _ g 

Low 0 - .'3 



ERIC 
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If >ou wish you may use tht scale in either of two ways, by using 



an X, 








I 1 










or by putting in 


a number* 








. .8 . 


"Certain" should 


be recn rded as 


follows ; 




1 1 


- t - -1 



MO I)b:LS AND SIMULATIONS 

The word "model" Is snmecimea used, to refer to a program that 
mighc be called a general^purpose subroutine in the sense that it, 
together with other modeia, may be run together in a simulation. Thus, 
many simulations may be specific "onS'-ihot" configurations of models 
operated only to answer a specific question. Sometimes a larga-^cale 
program may called a simulation and is used more than once with 
different inputs or even with models or subxoutinas added. If you 
feel that an important technical distinction concemlng your MSG has 
not been madej comment accordingly and nota where it might influence 
an answer, 

BUILDERS AND USERS 

The first saction of this questionnaire is oflented more toward 
those who use the outputs from an MSG applied to a particular problem 
or study than to those who designed or built the MSG. A user who is 
reasonably well acquainted with the technical aspects of the work may 
be able to answer the second section of the questionnaire* However, if 
it appears neceasary to have another individual or group answer the 
second part, this should be done and the second group should be 
identified. 

i : " i • 

ERIC 
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Part 1: Basic Information on Purposes, Use, Benefits, and Costs 



/ 



ERIC 
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IJOTK: Coding data^ pei?pons^ data^ and oth^r infomaticn }ioi :>i Lh' 
opiginal que&tzoyinair*i' appeaP t*: italics. 



*l-0 QUESTIONNAIRE Tim ( 



[Those Qutstions marked are of particular Importance nnci 
therefore extra care and accuracy are called forp] 



ill. Simulation/S tydy/Model and Author Name Tag: This calls for the 
name of the first or primary effort, list name of the agency or 
firm(s) that built the game, simulation or model (MSG), and the namu 
of the two or three major authors or designers* 

The agency or authority is the official group ordering the work. 
The sponsor calls for the name of the officer or official responsible 
for ordering the work, 

2-0 SIMULATION/STUDY/MODEL NA>E ( ) 



DESIGNER BUILDING AGENCY OR FIRM ( 



AUTHOR(S) NAME(S) ( ) 

AGENCY OR AUTHORITY (I ES) ( ) 

( ) 

SPONSOR(S) OF WORK: NA>E(S)( ) 



'ATF' 

* 


12, 3/10-11' 




Bespome 


3-0 


CATEGORY : 


MODEL 


46.2 


3-1 


CATEGORY • 


SIMULATION 


34 >S 


3-2 


CATEGORY : 


MAN-MACHINE GAME , 


8.S 


3-3 


CATEGORY • 


MANUAL GAME 


1 > I > li, S 


3-4 


CATEGORY : 


MATHEMATICAL ANALYSIS OR STUDY 




3-5 


CATEGORY : 


OTHER ( ) Specify 


.... 4.8 



AW' , , . , . . . . , . / 



'This information^ whiah is listed for various qmstions through^ 
out the qu&stiannairs^ indiaates th& aode nanie of the variable being 
s^died mid its position in the keypunch dBcki H&nae^ in this aade^ 
the oade name is CATEGl^ oaaupying aard 3^ aolums 20 and 2h See Ap- 
pendiz C for the aompletB oodebook* The r&sponseB are percentages of 
132. 

' ' No maponse, 
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H> Respondent's Rolet [Describe your role with reapect Co this 
game or simulation, e,g,, participant/player/f under/user/deslgner/ 
implementor/caretakers ete« ] 



•ESiiuLU S/ 12-14 












i-O RESPONDENT' 


•s 


ROLE- 


FUNDER OR SPONSOR 




14.-} 


4-1 RESPONDENT' 


•s 


ROLE: 


USER 




SI. 8 


4-2 RESPONDENT' 


's 


ROLEt 


DESIGNER OR BUILDER 




n.i 


4-3 RESPONDENT' 


'S 


ROLE: 


PLAVER 




0.0 


4-4 RESPONDENT' 


'S 


ROLE: 


CARETMER 




IS. 6 


4-5 RESPONDENT' 


'S 


ROLE: 


CONTROL TEAM OR REFEREE . 




J, a 


4-6 RESPONDENT' 


S 


ROLE: 


OTHER (Specify) 




..... 7.6 



[CO^mENT■ Give or Attach a Brief Job Description] 



5-0 LENGTH OF TIME YOtJ HAVE BEEN ACQUAINTED WITH 
THIS WORK ( 



6-0 IF LESS THAN 3 YEARS, LIST LAST TWO ASSIGNMENTS ( 

)' 



*7-0 RESPONDENT'S INFORMATION: NAtffi ( _) 

RANK, TITLE AND POSITION- ( ) 

(_ ) 



SPECIAL QUALIFICATIONS AND/OR EDUCATION KELEVAKT TO THIS 
TASK: CDescribs) 
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8- 0 MAJOR STATED GENERAL PURPOSE OF YOUR MODEL, GA^IE, OR SIMULATION- 

[If you were given a few minutes to describe the project to a 
senior officialp what could you say? (This could be one Co two 
doubles-spaced, typed pages, or If you have a reasonably sliort 
written summary , simply attach a copy*)] 

9- 0 SPECIFIC PURPOSES: 



e two sp^ecif ic examples of questions or operational problems 
this MSG has been used to answer *] 

[NOTE: If more thau one category appliesi circle more than 
one answer in your reply, ] 

PIO , Purpose: Hi is question is somewhat redundant wltli #8 and ?/9. 
Howeverj here you are asked to conform to a few-word description of the 
work. It is important to note the distinctions made in //10=3 and illQ-^ ■, 
Some strategic games have introduced diplomatic considerations and 
international b argaining. These would fall under the category ?flO~3s 
Other simulations and gaming exercises may be explicitly concerned with 
internal economic and domestic repercussions. These would more appro-- 
priately be classified under //10-4, It is of course possible to have 
exercises that may be classified under more thau one category, such 
as both //10-3 and //10-4. 

PUim, 3/17-19 

10-0 PURPOSE: TECHNICAL EVALUATION ^4.7 

10-1 PURPOSE J DOCTRINAL EVALUATION , . , , , 18.2 

10-2 PURPOSE: FORCE STRUCTURE EVALUATION 28.3 

10-3 PURPOSE: MILITARY -^DIPLOMATIC ANALYSIS/ INTERNATIONAL 

RELATIONS , . . , 0.8 

10-4 PURPOSE: MILITARY-POLITICAL-ECONOMIC ANALYSIS/DOMESTIC 

RELATIONS 0.0 

10-5 PURPOSE: TRAINING/EDUCATION _ , . . 

10-6 PURPOSE: RESEARCH METHODOLOGY J,^ 

10-7 PURPOSE: OTHER (Specify) J*. 6.1 

Low Middle High 'ertain 

lOCL, 3/20 , 3.0 I 3.8 , 13.6 i 6?JJ t 26. a i 

Confidence Le vs 1 



#11, Classification: Often simulations or models per se are unclas- 
sified, Howevers when military plaiining factors are added thev then 
become secret or may have an even highar classification. This means that 
for some there will be supportive docuTOnts that fall into many classl- 
ficatlons, #ll--5 Proprietary, We include business classifications, 
such as simulations run Internally by corporations and classified as 
corporate confidential; furthermore, proprietary implies that the 
documents are made available only by the proprietor to whomever he 
sees fit, 
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ijt 1 i ' 




ft, ^ 










11- 


■0 


CLASSIFICATION 


(WITHOUT 


INPUTS) \ 


NA; UNCLASSIFIED . . . . 




11- 


■1 


CLASSIFICATION 


(WITHOUT 


INPUTS): 


FOUO 


6,3 




< y 


CLASSIFICATION 


(WITHOUT 


INPUTS) \ 


CONFIDENTIAL , ; 


.. K.J 


11- 


'3 


CLASSIFICATION 


(WITHOUT 


INPUTS) : 


SECRET , . . * 


f. 

i i i .J , 1 . 


U- 


■4 


CLASSIFICATION 


(WITHOUT 


INPUTS) \ 


TOP SECRET , , , , 




U- 


5 


CLASS IFICA'^ION 


(WITHOUT 


INPUTS): 


PROPRIETARY , 




u- 


6 


CLASSIFICATION 


(WITHOUT 


INPUTS) s 


OTHER (Specify) 





12-1 
12-2 
12-3 
12-4 
1.2-5 
1.2-b 



CUSSIFICATION 
CLASSIFICATION 
CLASSIFICATION 
CLASSIFICATION 
CLASSIFICATION 
CLASSIFICATION 
CLASSIFICATION 



UNPUT DATA, OTHER) ^; 

(INPUT DATA, OTHER)^: 

(INPUT DATA, OTHER)^! 

(INPUT DATA, OTHER) ^ I 

(INPUl' DATA, OTHER) ^ I 

(INPUT DATA, OTHER) ^! 

(INPUT DATA, OTHER) : 



NA| UNCLASSIFIED 

FOUO 

CONFIDENTIAL . , , , 
SECRET t . 
TOP SECRET 
PROPRIETARY . . , . , 
OTHER (Specify) . 



J 



12 
0.0 

BJ. 1 
1U4 
0. 



* I DESCRIBE] 



//13* Professional Review. Wiiat professional reviewing procedures are 
ik^ed to check periodleally on the validity of this MSG^ its use and 
iiB iiiputs? Is tliere any formal reviewing group external to botli tiie 
users and producers? 

13-0 PROFESSIONAL REVIEW DONE? NO . 40, t 

' YES i i i i i » i "i^m , i # * 49 i 2 



m I 10.6 



BY WHOM? (Specify) 

ft 

REVDAT^ 3/26 | 

WHEN LAST DONE? { _) : [Jot done 40.9 

< 6 months * k * . 13,2 

6-22 months , 13, 6 

13-24 fnonths 6.1 

25^36 rnonths 4,5 

> 26 months . . , i . . « ^ # . . ^ 4,6 

NE ' 12.1 



iflA . MSG Parent or Antecedenti; The first three .subcategories 
ask for information on a formal model parent^ l#ei5 existing model, 
simulation, or game that is acknowledged as the direct parent of the 
niode'l categorized. Even for original models, there in general was a prior 
work or specific piece of literature or ieveral models and pieces of 
literature which provldad the suggestion or stimulation (positive or 
negative) for the construction of this model. 
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L';?.";/"..iff, ,3/se 

14-0 MSG PARENT OR ANTECEDENTS, UIRECT; NONE .'7 

14-1 MSG PARENT OR ANTECEDENTS , DTRECT; ONE ,!,?.. J 

14-2 MSG PARENT OR ANTECEDENTS, DII^CT: MORE THAN ONE 

(Specify) 20. t 

14- 3 MSG MODEL PARENT OR ANTECEDENTS, INDIRECT, NU>fflER ) J.d 

MR Low Middle High Certain 

14CLj 1/2? I flJ.fi . I j.i , 12.1 , 43. 3 , 18. j , 

Confidence Level 

15- 0 NAME DIRECT PARENTS ( ) 

( ) 

16- 0 NAME INDIRECT ANTECEDENTS- ■( ) 



#17. Development Initiation Date; 
nearest month or year. 

*17-0 DEVELOPMENT INITIATION DATE' 

HB. Use Initiation Date: 
18-0 USE INITIATION DATE: 

DEVTMl, 3/29 
ELAPSED TIME OF DEVELOPMENT (it 18 ~ 



Give date or approximate date to 



( ) 



#17): < S months 7.6 

3-6 months 19, ? 

7-12 months 21.2 

13-18 months 25. 2 

19-24 months 8,3 

86-30 months 5,3 

31-36 months 3,8 

37-42 months 3.8 

> 42 months ......... 3.0 

m 12,1 



This means the first production run, 

( 



#19* MSG Spinoff, This refers to niodelss games or simulations for 
which this MSG was a parent. We also count the same MSG used elsewhere 
by a different agency ^ authority, or group in which case ''same" would 
be entered as a nam of a spinoff. 

MSGPHO, S/28 

19-0 MSG SPINOFFt NONE 60,6 

19- 1 MSG SPINOFFS ONE OR MORE (Spaclfy nutabar) 1 16,7 

> 1 ly.i 
m ........ 5,3 

Low Middle High 

t > » » 

Confidence Level 

20- 0 N^S OF ONE OR TWO SPINOFFS: ( ) 

EKLC 
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HODEL/SIM ULATION/GAM PRODU CTION AND PURPOS E 

iHl, Funcllngi In some Instances, especially when individuals do a 
fair amQunt of work in their "ipare time," one ahould nQte more than 
ona funding source. Multiple fimding gources may also arise when a 
game is itarted at one location and completed, run, or used at another 
Ipcation. NSF would be classified as #21-1 and //21-9. 

WHOl, 3/30- 33 



21-0 


FUNDING 


SOURCE: 


C 


) 


, ... 0.0 


21-1 


FUNDING 


SOURCE- 


FOUNDATION ( 


) 


, . . . 0. 3 


21-2 


FUNDING 


SOURCE: 


PRIVATl (HUSINESS, SELF, 


MISC.) ( 


) L.,: 


21-3 


FUNDING 


SOURCE: 


ARPA ( 


) 


, ... O.lJ 


21-4 


FUNDING 


SOURCE: 


JCS ( 


) 


0,3 


21-5 


FUNDING 


SOURCE.- 


USA ( 


) 


... -12. 4 


21-6 


FUNDING 


SOURCE: 


USAF ( 


) 


... 27. 'I 


21-7 


FUNDING 


SOURCE: 


USN ( 


) 




21-8 


FUNDING 


SOURCE: 


OTHER DOD ( 


) 


^ , , 'I, A 


21-9 


FUNDING 


SOURCE: 


OTHER US GOVERNMENT ( 


) .. 




21-10 


FUNDING 


SOURCE: 


UNIVEF5ITY ( 


) 


• * m iJ ■ 


21-11 


FUNDING 


SOURCE: 


OTHER ( 


) 


... 0.0 



//22, Production*. //22^2 For profit includea the possibility that 

a game ia built by a for-^profit organization but not necessarily used 
as a product* For example^ it might be used for research or for 
internal training purposes. The producer may be a for-profit organlza- 
tinn building for the DOD. 

WHODIDI,. 3/34-3S - 



22-0 


MSG 


PRODUCTION: 


PRIVATE, INDIVIDUAL 


0. 


n 


22-1 


MSG 


PRODUCTION I 


UNIVERSITY , . , 




*i 


22-^2 


MSG 


PRODUCTION* 


FOR-PROFIT , , , . , , . . , 




0 


22-3 


mG 


PRODUCTION: 


ARMED FORCES , 




7 


22-^4 


MSG 


PRODUCTION: 


NOT-FOR'-PROFIT _ , _ 


22, 


0 


#23. 


MSG 


Initiation: 


#23--l Model builders/researchers refers 


to B 


I 



project where the original proposal was initiated with the individual 
specifically interested in researching and building the siifiulation , 
Much of MSG/research work falls under this category. The researchers 
propose the construction of the MSG to a funding or a sponsoring agency. 
W3-2 MSG/users may initiate a proposal for construction* For example, 
a decision may be made to run a model at an institution ^ich has a 
special facility. The request is made to the constructors and when the 
simulation is ready, those who requested it actually participate in its 
operation* 



//23-=3 ITinra may be a request made inteimally to an organl^atloti from 
a management group for the conatruetion of a model to be used by otiier 



In other words ^ 



requegts "b" to 



individuals In the organization, 

construct a model to be used by/'c." This Is a fairly coragn managerial 
structure. //23-4 An agency outside of an organization which intenda to 
use a model may request the organlEation to build It for the agency. 
For example, the local governmental group may declda to have ieveral 
sessions with an urban development alniulation. They may request private 
corporations to bulls it for them and may then uae the model for tralalng. 
operational, or advocacy purposee* #23^5 An outside agency may request 
a different institution to both build and use a iiraulatlon for their 
purposes. In this case, the sponsor is really only intereated In the 
results and not In the specific aspects of the tnodel. For example, in 
certain simulations or computerized battle models, the quMtion posed 
may be "What are the characteristics of this weapon imder a given sac 
of circuma tancasr' Tlie aponsor may approve having a simulation built and 
constructed| however, the sponsor^s technical interaat per se, is only 
limited to the results and not to its operation. 



23-^0 MSG INITIATION: 

23-1 MSG INITIATION: 

23^-2 MSG INITIATION! 

23-3 MSG INITIATION: 

23 ^ MSG INITIATION: 

23^5 MSG INITIATION: 



NA; UNKNOWN 

MODEL BUILDERS /RESEARCHERS 18,2 

MODEL USERS INTERNAL TO ORGANIZATION iOa 

NON-'USER, INTERNAL TO ORGANIZATION 10.6 

USER, E)CrERl^AL TO ORGANIZATION SS. 5 

NON^USER, EXTERML TO ORGANIZATION 12.9 



illk. Initiator Purpose: #24^1 The differentiation we wish to make 
between the terms "teaching'* and "training" li that training Is more 
concerned with "how to*^ whereas teaching is more concerned with 'Vhy." 
In many instances teaching and training blend imparceptlbly into each 
other. #24^2 Analysis t This meaning is to be distinguished from 
//24--5 research/ thaory development, Malysls means the honest grappling 
with a specific question or set of quastlona related to a given problem. 
The distinction is best made between operational modeling, where a purpose 
of analysis is fairly well known, and academic modeling, whera research 
and theory developTCnt are more the norm. 



INPUBPl, 2/3?^39 

24-0 INITIATOR PURPOSE: 

24-1 INITIATOR PURPOSE I 

24-2 INITIATOR PUM^OSE! 

24-3 INITIATOR PURPOSE i 

24-4 INITIATOR PURPOSE : 

24-5 INITIATOR PURPOSE: 

24-6 INITIAIOR PUBPOSEi 

24-7 INITIATOR PURPOSE; 



B4CL^ 3/40 



NA; UNKNOWN 

^ TEACHING/TRAINING , , . , 

ANALYSIS/DIAGNOSTICS 

OPERATIONAL ,,,, 

EXPERI^ffiNTAL 

RESEARCH/THEORY DEVELOPMNT 
POPULMIZATION, ^VOCACY 
OTHER ( ) , . , 



NR 
i 4.5 



Low 
4,6 



Middle 



High 

55,2 



Confldenca Level 



,f , 0,0 

US 

.... 81,8 
10,6 
• • • « 0*8 
.... S.3 

0,0 

.... 0,0 

Certain 

I 18,7 I 



ERIC 
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#25, Specificity of Purpose of Funding Sourci : The specificity of 
purpose of the funding source will in general depend heavily upon 
vhether or not the funding source was also the initiator of the game 
project. For example, a proposal may be made to ARPA to study uncon- 
ventional warfare. They will sketch out certain aspects of their 
proposal and more or less what they are going to do* The decision to 
fund this will be based upon whether or not the group appears to be 
competent arid the work seema to be "reasonable,'' and a considerabl*s 
amount of leeway will be left for the grou, 's actual work. 
Such a situation would fall under #25--2 mod'arately specified or #25-^ 3 
diffusely specified. On the other hand, there may be a specific request 
from a government agency to test a specific piece of equipment and to 
uae a study or simulation to report on the quality of this ciquipment. 

SPCPHF, 3/41 

25-0 SPECIFICITY OF PURPOSE OF FUNDING SOURCE t NA; UNKNOWN 75.'' 

25-1 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: TIGHT 

25-2 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: MODERATE 40.9 

25-3 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: DIFFUSE 14,4 

NR Low Middle High Certain 

2SCL^ 3/42 1 6.1 i 6,1 t 29.5 \ 42.4 j 15.9 \ 

Conf idenr.e Level 

#26* Best Alternative Procedures. Imagine that the objectives of the 
simulation must be achieved by a different means. It might appear that 
to check none and then to check something else would be mutually in- 
consistent* Howevar^ if the MSG is used for more than one purpose this 
could aaaily arise. For exainple^ in a MSG used for eKperimentation 
there may be no alternative for the experiment, yet the tnodel may also 
be used for teaching. In this case, lectures or case studies would be 
reaaonable altemiitlveg , 

ALTPHOl, 3/43-44 



26-0 


BEST 


ALTERATIVE 


PROCEDURE; 


NA; UNKNOWN ^ , 


i < • « « • 


. .... 10.0 


26-1 


BEST 


ALTEI^1ATI\^ 


PROCEDURE* 


NONE OR VIRTUALLY 


NONE 


22.0 


26-2 


BEST 


ALTERIMTIVE 


PROCEDURE: 


LECTURES , . < 




0.0 


26^3 


BEST 


ALTERNATIVE 


PROCEDURE: 


CASE STUDIES/HISTORY 


11.4 


26-4 


BEST 


ALTEFa^ATIVE 


PROCEDURE I 


ANALYSIS 




43. S 


26-5 


BEST 


Al.TEraATlVE 


PROCEDURE i 


E}ffERIENCE 




6,8 


26^6 


BEST 


ALl^W^ATIV'^ 


PROCEDURE- 


GAMING 




i « 4 i- I i W 




NM . . 










0.8 




Major Use of MSG; 


; Select the 


I appropriate catef 


sories 


from the 



list below, AJ.so provide written commentary to explain what you mean 
by the categorizes selected, 1,6,, give "for instances," 



27-0 


MAJOR 


USK 


OF 


MSG : 


NA- UNKNOWN 


? ■ 


27-1 


I-WJOR 


USE 


OF 


MSG: 


TEACHING/TRAINING 




2 7-2 




USE 


OF 


MSG: 


ANALYSIS/DIAGNOSTICS ... . 




27-J 


MAJOR 


USE 


OF 


MSG: 


OPERATIONAL 




27=s 


MJOR 


USE 


OF 


MSG: 


EXPERIMENTAL 




27-5 


MAJOR 


USE 


OF 


MSG: 


RESEARCH/THEORy DEVELOPMENT 




27-6 


MAJOR 


USE 


OF 


MSG- 


POPULARIZATION. ADVOCACY , 




27-7 


MAJOR 

Nh ,.- 


u >: 


OF 


MSG: 


OTHER ( ) ... 














m Low Middle 


High ''6rta'': 








^/4S 


1 S.O 1 3.0 1 14.4 1 


6S.3 , 1 



Confidence Level 



//28* Analysis Prucadui'ds: Explain, providing for instances , the type, 
amount, and rigor ui analyclc procedures used on the output of this MSG. 

Low Middle High 



Confidence Leve] 

#29. Judged Effectiveness of Best Alternative ProcGdure: This is 
for the main purpose of the MSG. By main purpose of the MSG we mean 
the main use that in fact has been made of it. Initiator purpose had 
the possibility of alternative procedures for more than one use, In 
answering this question we restrict ourselves only to the major use* 

29-0 JUDGED EFFECTIVENESS, BEST ALTERNATIVE PROCEDURE- ( ) 

Low Middle High 



Confidence Level 

f/30. Number of Briefings: Total number to date based on this MSG. 



BEIEF, 2/4^ 

*30-0 NUMBER OF BRIEFINGS : UNKNOWN V 

30-1 Nm^BER OF BRIEFINGS: (FvBclfy) 

0 , ■ 11. 

2-S 21,4 

^-6 

, c.s 

20^12 E.S 

, US 

2G-18 o,e 

> 18 , 



#31, Level of Briefing: By level of briefing we refer to organization 
and organizational level and a description of the personnel level involved. 
For example^ number and rank of generals or senior government officials 
present. 
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31-0 LEVEL OF BRIEFING^ [Describe] 



if32. Purpose of Briefing: Identify and state purpose of two most 
iniportant briefings^ i.e., what decisions depended on this MSG? 



^32-0 PURPOSE OF BRIEFINGi [Describe] 



ff33. Importanca of MSG to Decision: What impact did MSG have on 
decision just noted? Describa specifically. 



r34. Measure of Benefits ^ Specify what you ragard as a reasonable 
measure of benefits and iuccess from this MSG, 



MSC FROni'CTTON COSTS 



f/15. Direct Costs to Build: Under this uategon' we mean at'tual 
expenditurea of money earmarked for the purroses of developing this 
particular game, formal pay for working tiinft, and measures nttribtit.ed 
to the cost of, the game. In many instances where there is a ereat amount 
of informal work done, the direct cost for the development of the game 
may be zero, althouf.h fhe indirect and unallocated costs mav be enornsous. 
For example, the UCI.A business gnnia was developed liea\d.ly bv the nsc: of 
faculty time which was not partlnularly assigned to gdrne buildinv;, Manr- 
university ganies have this property. Nany games built ln-hou«H Co an 
institution without direct corftractual assistance aluo liavf th.is propen%-, 
Thus , the quesLirr Hhould be construed as one of findiuy out whuii diracL 
monies were assigned for tlie purpose of game cons t ruction . 

Cf'TDIH, 3/dJ 

35- 0 DIRKCT COSTS TO BUILD i (Specifv to nearest $10, 0001 

< $4^K , , ; ,'. 

o5U-oj:jK 

$100-ii^4?K -i. 

CSSC-i'K'iJr: . ^ 

$sOO-Hy4SK y. ' 

$750-lii^?9K j;!^ 

$1~$::.^9 million ^ ^: 

!^A~:i}ik?wM~^W 

■W^' Low Middle High Jex'taiK 

3/£l J \ i S.e I 20.6 i SO^S j_ 33.(^ | -'.i^ i 

Confidence Level 

/'36, llie question concerntng funding for development and building is 
specifically directed at the ac-counting question of what money has been 
formally iissigned to the efiort concaming a specific model,' simuJacion, 
or game... Tnus it will almost always be an underestimation of cost. 

IjI'-F^f. 3/S'j 

30-0 DIKECT FUNDING TO BUILD: NO ? 

36- 1 DIRECT FUNDING TO BUILD: YES j.-'n 

'i:^ 's.)- 

37- 0 .4M0IJNT OF FUNDING TO BUILDl (Specify) 

m Low Middle High Certain 

37 CL, 3/56 



36.4 y 8.S i 29.7 \ SB. 8 i e.J_ 



Confidence Level 



ERIC 



-98- 



//38. Total CosCs; Directs Indirect, Imputed, and Unimputed: The 
ranges are purposely kept relatively wide open because of the eKtreme 
difficulties in accurately Judging the costs. In some cases a variance 
of 100% or more is to be expected. This will probably be a rather 
surprisingly large number if one is honest with himself. There is an 
obvious political problem here* Many nf the unimputed costs would in the 
course of time have an opportunicy cost of zero. Furthermore, to a great 
extent many of the unimputed costs are extra hours of work put in bv 
oneself and not paid for. Hence they do not necessarily come out of any- 
body's budget or funding. They might be called the '-Out of Hide Costs/*' 

38-0 TOTAL COSTS TO BUILD^ DIRECT, INDIRECT, IMPUTED, UNI^£Pl^^ED: 
(Specify to nearest $10,000) 



ihO-idBK 14.4 

noo-mu 16.7 

^2S0-$499K _ , , , _ , 12. i) 

cSO0-$749K 3.8 

$75J'$$99K 3.S 

$l-$2.49 rnillian , 6,5 

$2,S^*)5 mllion , 0.3 

M-unk7Wu)n-NH , , . , , , , , 18. j 

NR Low Middle High Certain 

cSCL, 3/58 I IS. 2 , 26.5 i 37.9 i 15.9 \ 4.5 \ 

Confidence Level 



MODEL /SIMULATION/GAi-IE OPERATING COSTS MD OPERATIONS 



//39, Under costi to operate we include professional time, suppdrt 
time, set-up time, computer time, and experimental iubject or participant 
time together with overhead. In many cases many of these items are pro^ 
vided free. As we are discussing actual expenditures, the estimates we 
will be asking for will be gross underestimates. 

There is possibly more variability in operating costs than in bundl- 
ing costs for some cypes of games. The variability comes in the way 
player time is counted as a cose as well as facilities' use and operator's 
time. For example, with many businesi games played at universities It 
would appear that the financial costs are zerOj as the profejiorial and 
student time is not regarded as a direct cost and in some instances 
compucation does not appear as a directly imputed cost, and the use of 
facilities which would otherwise be unoccupied is deemed to be free. 
Any attempt to replicate that game in an envlronmant that does not 
have these features may be eKtremely costly. 

A political military exercise is usually run as an Individual affair, 
and we should consider further replications even though they T.ight be 
regarded as part of the same eKperiment as separate items. With educa- 
tional games or time-shared experiments, costs are calcuiated based on 
individual game or subject use. For simulation the cost to operate Is 
the cost to eKplore the answer to a typical question. This is relatively 
vague I but an uppir bound would be the amount of exploration that might 
result in a small separate publication. 
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CSTOPS, Z/59 

*39-0 ANNUAL COST TO OPERATE, GROSS: (Comment, if necessary.) 



Yaar Cost 
i9_ 

19_ ___ 

19 ' Specify for last five years, 

19_ ' 
19_ 

< $9K 23.fi 

nO-$24K 16.7 

$2S-$4$K 7.e 

$S0-974K 3.8 

$?S-$99K S.3 

> SIOOK 10.6 

NR 32,6 

NR Low Middle High Certain 

^9CL, 3/eO I 26,8 , 22,7 i 33.2 i 12.9 i 5.3 i 

Confidence Level 



HQ, Cost to Operate, Single Use: What does it cost for a single use 
of the MSGj assuming only minor or no variations In input values? For 
example, name such a use and date it approximately. By use, we mean 
for a single study effort. (This of course may vary, but give an 
average estimate.) 

C3TRUN, 3/61 

*40-0 COST TO OPERATE, SINGLE USE: 



< $9K 49.2 

$10-$24K 9.8 

$2S.$49K 2.3 

$50-$74K 0.8 

$?S-$9$K 2,3 

> $WOK 3.S 

NH 31.8 



//Al, Annual Update" Costs: If the model is under eontlnuoua dtvelopment, 
what are the annual costs of these activities, over and above "normal" 
operating costi? 

CSTUP, 3/62 



41-0 ANNUAL UPDATE COSTS, TOTAL DOLURSs ( ____) 

< $9K 27.3 

$20-$24K e,8 

HS-$49K 13,6 

$S0~$74K 2.3 

$?C'$99K l.S 

> $100K 2,3 

NE 46.2 



ERIC 
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41-1 ANNUAL UPDATE COSTS, PROFISSIONAL m'-YEARS .• ( ) 

41-2 ANNUAL UPDATE COSTS, PROGRAI#fflR MAN-YEA«J : ( " — 

m Low Middle High Certain 

41CL, 3/63 i_S7.o j 1 2. J , 27.3 i 36. 0 j r i 

CQiifldBnce Level 

Oil. Operational Life Span (to date); By operational life span we mean 
the period starting after developinent is coiViplete, beginning with the first 
operational use of the model to the period when it and its analysis or 
post-run Mposltlon are sat aside, having sBrved their purposfe. 

If a inodal, simulation, or game th.pt has baen In more or less continuous 
use is still in operation, both that length of time of operatlun mid the 
fact that it is still operaclonal should be Indicated. 

0PLIF2, 3/64 

*42-0 OPERATIONAL LIFE SPAN (TO DATE) : (Specify) 



< 3 months 9,2 

3-6 mnthe , 

7-12 montns , , 

23-18 months S.S 

19-24 months ............."..* k' i 

25-30 months 5, ,9 

Sl-SB months; , , 3^3 

37-42 months ^9*5 

> 42 Tnontha 23.5 

^ 2B.B 

NR Low Middle High Certain 

42CL, 3/6S I 12.1 | 3,8 j IB. 2 , 38.6 , 27.3 1 

Confidence Level 



/M3. Still Active?! By this we mean, "Is the MSG in its original or 
mildly modified form still actively being used for producclon runs?" 
We contrast this with a seri&us revision that has resulted In uither 
a new name or an appellation such as Mark II, or Mod III, and so on. 



ACTIVE, 3/66 
* 

43-0 STILL ACTIVE?! NO 27.3 

43-1 STIli ACTIVE? : YES ' f?S."a 

^ .....llMllllll]], y.'e 

NE Low Middle High Certain 

43CL, 3/67 \ 9.8 \ 0.8' ■ 6,1 , 60.8 1 32.6 , 



Confidence Level 
*44-0 MODEL USERS-. 



[Specify all agencies who directly use the MSG outputB by full 
name (not mllltar/ abbreviations); indicate the prime user.J 
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/;45. Operational Use: Annual Frequency: By this wa are trying to 
find Out how many times this MSG is actually exercised or run annually, 
Specify for the lait five years. 

Year Number of Times 

19_ 

19__ ' 
19_ ' 
19_ . ' 

19 



OPFRQl, A/ 63 
Not opamtd-onal 
1-5 

e-io 

12-15 

16-20 ,,,, 

81-25 ...... 

'26-50 

51-200 

> 100 

NH 



16,7 
23. 6 
6.8 
4,5 
1.5 
3.0 
7.6 
9.1 
IS. 9 
11,4 



EXPUSE, 3/69 
46-0 EOT ERIIffiNTAL USE : NO , 
46-1 EOTEMtENTAL USE: YES 

m , 



I « i a i a « 



76,5 
17.4 
6.1 



#47* Experimantal Example: If 46-1^ i.e., "yeij" give a for InstMict, 
[Describe] 

#48. Experimentsl Purpose, Initial: Was thli MSG d^signtd Qriginally 

for experimental use? 

EXPRP, S/70 

48=0 EOTERimNTAL PUWOSE, INITIAL; NO 88,4 

48-1 E)a>ERI>ffiNm PUOTOSE, INITIAL: YES * 9.8 

«*iiiiii>*iBBiiaiaia^a9a*»caaa«9a*i>cC>t>9!VV!l!l«llci S 9 8 



EDUSE^ 3/71 

49-0 EDUCATIONAL USE: NO 

49-1 EDUCATIfflJAL USE: YES 

m 



llllltfaiti 



f i • a a i t 91 # 7 

6.3 
3. 0 



ERIC 



#50* Educatienal Examplei If 49^1^ l*e*, "yei," glva a for Instanes, 
[Describe] 

#51 i Educational Purpose, Initial i Was this MG dtsigntd originally 
for educational use? 

EDPRO, 3/72 

51=0 EDUCATIONAL PURPOSE, INITIAL; NO 9U7 

51-1 EDUCATIONAL PURPOSE , INITIM. i YES US 

iiiiiiif*|ffiiiia!9ft«iiaiii9ii|i«fa!iaiiii!iitif«liil 6m 8 



#52, Transfarablllty of MSG Use: #52--0 is a model not intended for 
transfer* ^ example of such mighc be a claisified simulation run to 
test for some particular parameter value and diipensed with after the 
runs I of a study or simulation may be extremely classified wl th onlv 
oiie user in mind<i //52^1 This would be something like chess or Go wliere 
once the rulei have been transmitted, one could take a piece of paper, 
draw the board^ obtain some stones, and play. In general, a giim^ of 
this variety cm be transferred immediately at little or no cost. (^52*2 
fi^ exan^la of a game in this category would be a ganie that Is not overly 
compleKs is extremtly well documented and produced ^ and is generally 
available. The games Sitnaoc or Whifn^proof or Suffimit or Democracy would 
all fit into this category* 

52=-3 Middling would cover simulation games such as the UCLA business 
game where the size of the program is not enormous and computer require- 
ments are such that, at least in the United States , many institutions 
would have the facilities. Furthermore j the documentation is reasonably 
good* To get it operating^ such a game would require one or two months ^ 
taking into account telephone calls , time delays in the malls, etc. 
52-^5 Simulations that are extremely difficult to travel are ones that 
depend upon specific facilities and crews of eKperlenced individuals or 
that are enormous in computational size. For that matter they may 
not be cQmputer games ^ but have become so large that they should be 
regarded more as institutions rather than games. Examples of such are 
METROj The Rand Logistla Lab experimenti, TE>ffER, and the Carnegie Tech 
game. The reaions why these cannot travel are different; however ^ in 
each case the amount of work required to transfer the operation 
is enormous. 



TRANSU, 


, i5/ 7 3 










52-0 


TRjU^S FERTILITY 


OF MSG 


USE: 


NOT INTENDED FOR TRANSFER , 


. 10. € 


52-1 


T^SFIR^ILITY 


OF MSG 


USE: 


GENERALLY 


. 28. a 


52-2 


TR^SFERABILin 


OF MSG 


USE: 


MODERATE DIFFICULTY , . . 


. 17.4 


52-3 


TRANSFERABILITY 


OF MSG 


USE: 


MIDDLING DIFFICULTY 




52-4 


TRANSraRABILITY 


OF MSG 


USE- 


DIFFICULT 


. 18.2 


52-S 


TRANSraL^ILITY 


OF MSG 


USE: 


EXTRME DIFFICULTY 


. 7.C 




NR 








. 1. £ 



[Describe whether transferability just indicated pertains to any of 
the following situations: (1) use by another person or organization at 
a new site, (2) use by the same developer/designer/builder at a new site, 
or (3) use by another person or organization at site where MSG presently 
operates i ] 



//53, Transferability: Costs to 
in //52, please estimate how much 
norMl operating costs, would be 
this MSG. 



Operate: Using information provided 
additional amount, with respect to 
required to transfer and then operate 
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TMMSC, 3/74 



53- 


•0 




TO 


opfratf » 

W 4 £i An £* ■ 


) U rtll w W ii 






53- 


1 


TRMSFER COST 


TO 


OPERATE: 


COST , , , 




.... 10.!' 


53- 


2 


TRANSFER COST 


TO 


OPERATE: 


COST-COST+10% . . . . 




.... J£.P 


53- 


3 


TRANSFER COST 


TO 


OPERATE: 


COST+ln%-'COST+25S 






53- 


4 


TRANSFER COST 


TO 


OPERATE- 


COST+25%-COST+50Z 




• • » k tJ . L 


53- 


5 


TRANSFER COST 


TO 


OPERATE t 


COST+50%-COST+100% 




...» l" . 


53- 


6 


TRANSFER COST 


TO 


OPERATE t 


> COST+10Q% . . 




* * * . £ , r 










m 


Low Middle 


High 


Certain 












B \ \ 28.8 




1 10.6 , 












Confidence 


Level 




m 




Obsolescence i 


Comment on the speed at which 


you expect this 



MSG to become obsolescent , indicating the reasona why* We are not 
referring to the need for reprogramining for new hardware or for minor 
modif ications but to the state where it is no longer sufficiently 
relevant that either a major modification has to be made or it is 
compietely abandoned. For example j a special simulation may be built 
and run once for a specific purpose. A simple model may be used for 
many years, as long as the type of damage calculation it performs 
is relevant. 

#55, Related MSG: Does there exist an MSG that might be regarded as 
serving approximately the same purpose as yours? 

PUPMSa, ^/?6 

55-0 RELATED MSG- NO 62,3 

55-1 RELATED MSG: YES 4S,9 

m , , 3, a 

//56. Duplication of Use* If 55^1, i.e. "yes," name the MSG(S) 
and state why or why not one MSG could serve the purposes of all* 
If your answer is "nOs** i*e,s 55^0, state why you believe there is 
no MSG similar to yours. 

//5 7* Clearing House i Is there enough communication in DOD among 
different gaming and simulation studies or would a clearing house 
or. central agency be of use? 

CLEAR, S/7? 



57-0 


CLEARING 


HOUSE: 


HIGHLY USEFUL i... < 




. 14,4 


57-1 


CLEARING 


HOUSE; 


USEFUL 




. 43. P. 


57-2 


CLEARING 


HOUSE: 


SAME AS IS 






57-3 


CLEARING 


HOUSE ■ 


HARMFUL 




. a. 3 


57-4 


CLEARING 


HOUSE 1 


HIGHLY HARMFUL 




. 5. ,5 



M 5.^ 



Comments on Question //57 are required: 

O 

ERIC 
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#58. Standardization: After techniques and studies have been in 
existence .for some time, standardization and exchange of Common 
routines is extremely useful. However, sometimes premature attempts 
to standardize do more harm than good. In particular, redundancy to 
an outsider may not be redundancy to those doing a itudy. What is 
your belief in the advisability of increasing DOD gaming and simulation 
accivity for standardization? 

STNRDl, 3/78 
* 



58-0 STANDARDIZATION I 
58-1 STANDARDIZATION- 
58-2 Sm'DARDIZATION: 
58-3 STANDARDIZATION: 
58-4 STANDARDIZATION I 
NR 



HIGHLY USEFUL 

USEFUL , 

SAm AS IS 

HARMFUL 

HIGHLY HARMFUL 



6. 8 
24.3 
22. 7 
31.1 
10,8 

4. 5 



Comments on Question //58 are required^ 



Low 



Middle 



High 



' i J— J 

Confidence Level 

it59. Regional Centers: Models, simulations, and games are operated 
and built at many locations using many languages and different staffs 
and equipment. Would, for example, 3-5 appropriately cleared regional 
centers for most of these activities be a more or less effective way 
of supporting them? Discuss. 

REGCEN, 3/79 

HIGHLY USEFUL ■ g^g 

USEFUL 12,9 

SAME 8,3 

"ARMFUL 4e^2 

HIGHLY HARMFUL 24.4 

^ , 17. 4 

Ho, External Review Board' Would you conanent on reasons both pro 
and con having an external review board eonsider this and other MSG's 
built/operated/used by your organization or activity? Discuss. 

EXREV, 3/80 

HIGHLY USEFUL o.8 

USEFUL 14,4 

^AME 26. 9 

"ARMFUL 3g,4 

HIGHLY HARMFUL 

^ ■ • * - . i i i t ! i i i * . , , , , , * . , , , , , 22, 2 



#61. External Review Board, Specif ieationi Would yeu characterise 

a "good" gKternal review board as to composition and function? Discuss, 
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Part 11^ Model/Simulatlon/Game Characterization and Description 



NOTE: If Part II not filled out by Office of Prime Responsibility 
and/or userj please indicate who filled out Part II, 

Naine _ 

Organisation 

Relationship to 
Prime Office of 

Responsibility __________ 
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f?62. Scenario Type: f^62-0 means chat the inputs $ outputs, and incer^ 
pretation of the outputs are all numerical (example: number of Eargets 
destroyed)* 1.^62-1 means that a verbal description of the scenario must 
be available prior to use (example! a man-machine or free-form game). 
f?62-2 means that the Interpretation of an output is quali tatively 
modified or interpreted prior to being used (example: a written 
assessment of qualitative aspects of target damage may accompany a 
computer output), 

7/7 

62-0 SCENARIO TYPE: ONLY NUMERICAL , ^.^.i? ' 

62-1 SCENARIO TYPE: VERBAL DESCRIPTION NEEDED FOR USE i '. 6 

62- 2 SCENARIO TYPE: VERBAL DESCRIPTION NEEDED FOR ANALYSIS ,. ,! 

?;63, Scenario Description: Rich "realistic" may be used to refer to 

a scenario which is both rich in detail and purports to be a realistic 

description of some phenomenon* For example ^ some tactical games may^ 

go Co great lengths Co have a raalistic description uf weather conditions ^ 

troop morale conditions, terrain conditions, details on buildings, and 

so forth, llie measure of the description of environment should be 

relative to the real-wprld phenomenon being modeled. For example, a 

business game might have fewer details in it than a diplomatic game 

but be a richer model in relation to actuality than the diplomatic 

game. Furthermore, some games may have underneath them a mathematical 

model which is noc necessarily apparent to the users. The word ''Imaginative" 

can be used to refer to nonfactual modeling where a scenario may contain 

counterfactual or futuristic features. These aipects may be mixed in 

with other environmental categorizations, 

63- 0 SCENARIO DESCRIPTION: [Describe] 



??64» Mathematical Sophistication of MSG: #64-0, None equals less 
than high school math needed to interpret output or to participate as 
a player if it is a game. #64--l, Slight equals high school math. 
//64-2, Moderate equals needs college level math (BA) or engineering 
digree, //64-3, High equals requires an advanced degree to interpret 
the output* 

MATHS, 4/3 

64-0 MATHEMATICAL SOPHISTICATION OF MSG^ NONE 0.8 

64=1 ^MHEMATICAL SOPHISTICATION OF MSG: SLIGHT 25,5 

64-2 ^^ATHE^^TICAL SOPHISTICATION OF MSG: MODERATE B9,l 

64=3 MATHEMATICAL SOPHISTICATION OF MSGi HIGH 8,3 

MR P ? 

Low Middle High 

4. 



Confidence Level 
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//65. Timing of Moves: Event timing Implies that moves depend upon 
a specific event having occurred* Fixed clock timing impliei that there 
is a certain increment of time upon which model aetivities are based, 
lliere are some models that are both event and fixed clocks in the sense 
that generally the clock moves foTOard at regular periods, 6*1*^ descrip^ 
tions of gross national product in an International model. However, 
simultaneously there may be moves that depend on specific events, such 
as conditional checking for threshold effects ^ time in queues ^ etc* 

MuVES, 4/9 

65-0 TIMING OF MOVES: NA; UNKNOWN 21.2 

65-1 TIMING OF MOVES: EVENT , 39.4 

65- 2 TIMING OF MOVES: FIXED CLOCK 22.0 

'VMBTMTIJ^J 6S-1 and vB-:: 15.2 

tJR 2,3 

//66. Model Time to Real Time Ratio i In describing the ratio of model 
time to real time one has tiie problem of distinguishing between the 
period assigned the real time nnd the amount of that time which would 
have been used for the decisionmaking. For examplej.in a model in which 
the real time is meant Co be quarters, the price decision in a market may 
only take a week or two to make. In the exercise , twenty minutes may be 
allotted for the decisionmaking. We now have the problem of deciding 
whether to scale the twenty minutes against the one week or the three 
months. We suggest scaling against the allotted real time, l,et* the three 
months , 

MT/RT, 4/10 

66- 0' MODEL TIME TO REAL TIME MTIO: (Specify) 

m-UNKNOWN , 25. C 

HIGHLY COMPRESSED , , , , 14.4 

COMPRESSED " \ 34.1 

REAL TIME , 12,1 

EXPANDED ...,..V, 7,B 

HIGHLY EXPANDED 2.5 

NR 5,3 

NR Low Middle High CBPtain 

eeCL, 4/12 I il. j I 14.4 i_ U.8_ 1. 37 A i 11.4 i 

Confidance Level 

//6 7* Time Repreientadt In some Instances where the ilmulator is 
not necaiiarily meant to represent any specific structure the time 
period repreient^d might be interpreted as the present, but it might alio 
be batter to descrlbt it as unspecified. In cases of doubCi It Is best 
to note the model in both categoriea, 

TIME, 4/12 



67-0 


Tim REPRESENTED.' 






67-1 


rim REPRESENTED: 






67-2 


TIME REPKSENTED: 






67-3 


Tim REPRESENTED: 






67-4 


TIME REPRESENTEDi 
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#68. Leval of Reiolution, Model Time: This is the amalleit time unit 
recognized by the game, 

LRTIME, 4/23 

68-0 LEVEL OF BESOLUTION, MODEL TIME- NA; UNKNOWN 18.9 

68-1 I^VEL OF RESOLUTION, MODEL TIME: SECONDS 38.6 

68-2 LEVEL OF SESOLUTION, WDEL TlJffi: MINUTES 21.2 

68-3 LEVEL OF EESOLUTION, MODEL Umi HOURS 10,6 

68-4 mWL Of MSOLUTION, MODEL TIM:: DAYS '. 8.8 

68-5 LEVEL OF RESOLUTIOT, "MODEL Tllffi- WEEKS 0>8 

68-6 LEVEL OF HESOLUriOT, ^DEL TiJffi: QUARTERS 0.0 

68-7 LEVEL OF RESOLOTION, MODEL Tim: YEARS 0.0 

68-8 LSm C? RESOLUTION, >fODEL TI^: > YEARS 0.0 ' 

m d.o 

Low . Middle High 



Confidence Level 



#69. Level of Resolution, Space: In military games, the spatial level 
of reiolution is frtquently inportant; in most business games spatial 
level of reBolutlon is at beit crude. #69-4 refers to the situation wher 
datail may be iupplied for specific locations, but no detail is given 
between them: for exanple, details of the terrain around enemy airports, 
but no detalla for terrain between them. 



LRSPCE, 4/14 
69-0 aVEL OP RESOLUTION, 
69-1 LEVEL OP RBSOLUriON, 
69-2 mWL OF RESOLUTION, 
69-3 LEVEL OP RESOLOTION, 
69-4 LEVEL OF EESOLUTION, 
NR 



69CL, 4/1 S 



SPACE: 
SPACE: 
SPACE : 
SPAGE -. 
SPACE : 

NR 
I 8.3 



■i«iii>BiBiiBisilliBiii 

SMALL AREA (METERS) : 

MODERATE (KILOMETERS) 
LARGE (THMTER/CONTINENT) 
VARIED , * * i * , 



9.8 
34.8 
26,5 
12.9 
13.6 

2.3 



Low Mlddla High Certain 
\ 2a I 12.1 I . _43J_ I. 



Confidence Level 



#70. Level of Reaolutlonp Sides ^ In some models for some purposes there 
la no need to regolve the nature of individual teams • Gross performancas 
of the interaction, as a whole are baln| conslderad regardless of team slz€ 
For other purposes the saM model may be used with considerable attention 
paid to the team structure. 

Furthermore, a distinction between structiirpd and unstructursd groups 
must be made, In some inetancss, e»g.| when studies of the emergence of 
leaderBhip are being conducted, it is Important that no structure be 
placed on the taams, In other cases the teams may be given a structure 
such as that of a corporation or a military command* 



inO-y > As platooni dlvisioni air force, etc, vary in size batwaen the 
aervices, name the generic term for the unit. 
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^ ;j r n £7 


, 4/1 e 








70' 


-0 


LEVEL 


OF 


RESOLUTION, SIDES: 


lo- 


-1 


LEVEL 


OF 


RESOLUTION , 


SIDES : 


in 


- L 


LEVEL 


OF 


RESOLUTION, 


SIDES 1 


70- 


-3 


I FVFT 




RESOLUTION, 


SIDES; 


70- 


-k 


LEVEL 


OF 


RESOLUTION, 


SIDES: 


70- 


'5 


IJVEL 


OF 


RESOLUTION, 


SIDES • 


70- 


-6 


LEVEL 


OF 


RESOLmiON, 


SIDES I 


70- 


-7 


LEVEL 


or 


RESOLUTION, 


SIDES : 






NR ... 









NA 

INDIVIDUALS . 

SMALL GROUPS (STRUCTURED) . 



• i ■ ■ ■ 



30 a 

13.5 
19, 7 



ORGANIZATIONS, SMALL (< 100) 13.6 
ORGS,, large' (> 1000) 

ORGS . , VERY LARGE , 

SMALL GROUPS, UNSTRUCTURED 
NAM U"NIT ( ) ' 



Comment : 



Low 



Middle 



4.5 
3.0 
0.0 
3.0 
2.3 

Hlfh 



-L. 



Confidence Level 

#71. Level of Resolution, Military Action: The categorlea her* ire 
arranged in order of progressive genirality, thua #71-5 includei war as 
a part of ongoing diplomacy. 



71-0 
71-1 
71-2 
71-3 
71-4 
71-5 



4/1? 

LEVEL 

LEVEL 

LEVEL 

LEVEL 

LEVEL 

LEVEL 



OF RESOLUTION, 
OF RESOLUTION, 
OF RESOLUl'ION, 
OF RESOLUTICW, 
OF RESOLUTION, 
OF RESOLUTION, 



MILITARY 
MILITARY 
MLITARY 
MILITARY 
MILITARY 
MILITARY 



ACTION; 
ACTION : 
ACTION ! 
ACTION I 
ACTiro : 
ACTIOT: 



NA; UNKNOM 
INGAGMINT , 

BATTLE 

CAMPAIGN . . , 

DIPLOMATIC . 



19, 7 
44,7 
10.6 
11.4 
13,6 
0.0 



Coiiment: 



[Note specific details of thla MSG.] 

M Low Mlddls 

71CI, 4/18 I 6.8 I 3.8 



High Certain 



I 13.8 I 48. B i 27.3 \ 
Confidence Level 

#72, Random Events r It la posalble to mm a TOdel occiaionally with 
random events and on other occasiona without random tvtnta. In thii 
case both cateiorlia should be cher.ked. One should not regard this 
question as a binary choice; both are poislble. 

RNDM^ 4/19 

72-0 RANDOM EVENTS i NO 46. B 

72-1 RANDOM E7INTS i YES , B2a 

NH J, ,,,,,, 1*6 



Cowmmti [Note sptclfic details of this MSG.] 
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CHARACTERIZATION. PLANNING FACTORS AND DATA 



#73. Data Squrces and Validity. Where did the data for this MSG come 
from? List sources in as complete a fashion as possible. Were anv 
Indeptndent checks perfornied to insure the accuracy, timeliness, 
consiitency, and overall quality of the data? Describe them. 

yms, 4/20 

'73-0 DATA SOURCES AND VALIDITY: [Comments] 



NA; UNKNOWN 




MILITARY; NO CROSS CHECK 


, 30.0 


MILITARY; CROSS CHECK ' 




CIVIL 


hi' 


GENERATED OWN 


2a 


FIELD EXERCISE 


l.B 


COMBINATIONS , , 


25.0 


NE , , , . , 


t'.l 


Types of Data; It has been suggested that three 


types of data can 



be dli tlnguished In gamei and simulations ^ 
Typa 1 data - High certainty data 

Examples: range of a wtipon under specified CDnditionSj the 
size of a unit of troops^ etc. 
Type 2 data ^ Certain level of uncertainty 

Examples I outcome of a company fight j radar detection range 
(these need parametric studias and sinsitivity 
analysis for validation) , 
Type 3 data - High uncertain and hard to test 

Examplai : diplomatic behavlorj enemy goals, broad social 
or economic reactions to policy, 

'74-0 TYPES OF DATA 

[In tema of the three types of data requirements describe the data 
needed for'your game or simulation, ] 

#75. Number of Inputs (Conatanti, Parameters, and Variables) in MSG: 
Ihis may vary from use to ust| thirefore, if necessary, give lower bound, 
average, and upper bound, 

75-0 NUMBER OF INPUT CONSTANTS^ (Specify) 
75-1 NUMBER OF INPUT PARAMTERS: (Specify) 
75»2' NUMER OF INPUT VARIMLES^ (Specify) 

Comient i 

Low Middle High 

L_ I 1 

Confidence Level 
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//76, Nuniier of Output Variablei in MSG^ 'ttiis may vary from use to 

use I therefore^ if necGssary, give lower bound, ayeragej and upper bound, 

76^0 NUMBER OF OOTPUT VARIABLES: (Specify) 

Low Middle High 

Confidanct Level 

Cominant i 

nil. Intangibles: Are sometimes ruled out by limiting the scope of 
the study; by obtaining rulings from higher authority as to how they^ 
are to be treated; by using expert estimates; by usin| high and low 
bounding procedures or by other methods, 

77- 0 INTANGIBLES 

[Deicribe how intangibles are traated in this MSG. In answering 
give a for Instance.] 

f?78. Sensitivity Analysis j Discuss the importance of sensitivity 
analysis for this MSG and describe how It is done^ if it is done at all. 
We are not interested in sensitivity analysis done as a routine matter 
of debugging; rather^ what has been done since the MSG has been 
operational? 

SENSET, 4/21 

78- 0 SENSITIVITY ANALYSIS PERPOR>ffiD? \ NO , ....44.7 

YES 46.2 

m s,j 

Comment t 



79-0 DATA COLLECTION limi ( ) 

[Estlmati (in man-years) data collection time required,]" 

Low Middle High 

i . I i 

Confidence Level 

#80* Data Validations Frequently all data come from another agency 
or source with no checks from the user |roupi , Sometimea a user group 
obtains its information first hand by meaiurement, observation, field 
testis etc. Describe how you get your data inputs and what Independent 
checks or procedures you perform to challenge the validity of the daca. 
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DATAV, 4/22 
80-0 DATA VALIDATION: 

NA; umom e.8 

HIGH QUALITY ' ' IS. 6 

MODERATE ' 28.0 

WEAK , 29,6 

NOT DONE u,g 

m e.i 



MODEL OPERATION; SUPPORTING FACILITIES 

#81, Computer Used for Running a Simulation: We mean the different 
computers for which this model has been run. In some cases there may be 
only one, and in other cases many modifications may have been issued 
for different machines. List not more than the three most frequently 
used operating systems, 

81-0 CO>ffUTER USED FOR RUNNING: NA 

81- 1 CO>ffUTER(S) USED FOR RUNNING: THREE OR LESS (Specify) 

Low Middle High 

L L --^ i.^^.,..^ 

Confidence Level 

//82. Program Language' This calls for the language in which the 
simulation has been programmed. Frequently there may be a serias of 
languages. All should be noted, 

LANG, 4/U 

82- 0 PROGRAM LANGUAGE' NA 

82-1 PROGRAM LANGUAGE(S) : (Specify) 

FORTRAN 79.5 

COBOL ^0,8 

GPSS 3.0 

SIMSCHIPT 3,8 

ASSEMBLER 3,0 

0 0^ ifiij ■■■•■•■•••■■■■■■■■iaiciaiiiatlgitiiillftctiii ■■ 2 m S 

jy/? * » . , , i * » i i i « « i 4 5 

Low Middle High C&y^tain 

82CL, 4/24 I 5,3 i 0,8 | 6.8 [ 44J _\ 42.4 \ 

Confidence Level 

#83* Program Size: Approximately how many instructions are there in the 
language(g) noted above? 



MSGSIZ 

83-0 PROGRAM SIZE; (Specify language) 
83-1 PROGRAM SIZE: (Specify language) 



NAj unknom , !M 

< 2000 insiruoHonB , i^,;-] 

1-2K ;.. .r,^ 

^-4K 7^.u 

4-6K 

6-8K 

8-lOK s.O 

> lOK 27. :i 

NR ]>].] 



//84. Facilities r //84=4 This refers to the situation where a special 
system set of languages or program may have been written to accompany 
the running and general handling of a specific model. For example, some 
models depend upon the availability of much of the specialized extra 
hardware and software. Although it is possibli that the models them- 
selves can travel, much of their power is lost when the accompanying 
periionnel and equipment are not available. 



FACIL, 4/26 

84-0 FACILITIES: NA; UNKNOWN 6&.2 

84-1 FACILITIES: SPECIAL BUILDING OR INSTITUTION 2.P 

84-2 FACILITIES: DEDICATED COMPUTER (UNCLASSIFIED) 6.8 

84-3 FACILITIES: DEDICATED CO>ffUTER/CLASSIFIED TAPES 9,8 

84-4 FACILITIES! SPECIAL LANGUAGE, LIBRARY OR COIffUTATIONAL 

SYSTiM 12,9 

m 3,0 

Low Middle High 



Confidence Level 



SIMULATION/STUDY/MQDEL PRODUCTION COSTS 



General caveat on building costs. The possibilities for obtaining 
close cost estimates for many MSGs are difficult in the extreme, This 
is not merely a problem of gathering information; It is a problem of 
correct conceptualization of the costs that should be imputed to certain 
forms of work. In particular, joint costs play an enormously importanc 
role; thus it is not an easy matter to impute, costs for items such as 
computer time, joint use of educational facilities, and so forth. In 
this coding scheme we wish to stress that the costing figures presented 
should be used with extreme caution. 
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//85. Uevalopment Time is a concept about which it is difficult to 
be both precise and accurate. For our purposes, we must eniphasize that 
the categories indicated are crude in the extrenie. We are trying to 
indicate the elapsed time between the decision to build a particular 
model and the first production run of that model. In many cases after 
a model has been used once, development goes on for many years Thus 
our criterion may be regarded as presenting a iross underestimation of 
development time. Furthermore, additional complications appear on occasion 
as a model develops and it may change its name. A further clarification 
of this idea, according to our tneanlng, is the time from the inception of 
the work on construction until the first production run. This is con- 
trasted with a debugging run; they are not the same. We specifically 
do not consider further modification after the first production run has 
taken place, even though ex post faato, the first production run is then 
regarded as "axperimental. " 

D!;V7M2, 4/28 

85-0 DEVELOPMENT Tllffi- ( ) 

< 3 months 7 35 

S-8 months _ 22 0 

7-12 months , 25' 8 

IS- 18 months '' q ~g 

19-24 months ^ ! V 3 

2S~S0 months S B 

31-26 months 3' 0 

^7-42 months 3 0 

> -IB months - - - c' t 

"■^ 25.2 

m Low Middle High Certain 
8SCL, 4/29 I UA , 3,3 , 29, S 1 40,2 | 9.8 1 

Confidence Level 

#86. We contrast total man-years with profassional man-years . Under 
category //86 we include graduate students, secretarial help, program- 
ming assistance and any other forms of voluntary contribution of 
time. These are direct tnan-years and do not include allocation of 
Insticutional administrative overhead. 

DFMYl, 4/30 
* 

86-u DEVELOPbffiNT TIME- TOTAL MAN-YEARS- ( ) 

M- unknown .7 , IZ 9 

0-1 man-years .' 2l' 2 

2-6 man-yeare 32 8 

6-10 mm-yeara , g g 

11-20 man-years , ^ ^ 2 

> 80 man-years g j 

^ • sis 

NR Low Middle High Ceptain 
86CL, 4/31 I 18.9 | 21,4 , 41.7 1 S2,0 \ e.l | 

Confidence Level 
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#87* Professional man=years used in the development of a modal* Under 
this category we include both professional dtsigners and consultants. 
In many cases there are also graduate students, additional helpers * 
ordinary programining asslitance, as well as an enonnoua amount of offic 



staff. 

87-0 DEVELOPI^NT TIME: PROFESSIONAL MAN=YEARS: (_ J 

NA; unknom 22,1 

0-1 man-yeara 25,8 

2-6 mmi-yea^B , , * 22,6 

lUlO man-yeaP8 , . 9,8 

11-20 mn-^ysars 6.8 

> 20 nan-years , . 2,3 

/V/f \, 10,6 



//88. Development Teani Profeislonal Profile: Describe the professional 
makeup of the developnient team (including consultants) i 

*88-0 DEVELOPJffiNT TEAM PROFESSIONAL PROFILE: (Describe) 



NR Low Middle High Certain 

88CL^ 4/33 i 23,5 i 6,8 \ 20,5 i 37.1 i 12,1 \ 

Confidence Level 

DVPRGl, 4/34 

89-0 DEVELOPItENT TIbE; PROGRAI^R MAN- YEARS ( ) 

NAj unknom , . , , ,,,,,,,,,,,, i , * t 19.7 

0-1 man-yBars 37,2 

2- S man-years , , , 24,2 

6^10 man-^years • , 5 

11^20 man-ijeaps 3,0 

> 20 man-^yec^B . , , US 

NR ... 9.8 



MODE L CHARACTERIZATlONi DOCUMENTATION 



#90. DocuMntationi /^90-l Excellent means that the docimentation Is 
sufflciantly good that it can be picked up aliawhert by a dlfftrant 
group of people and operated without or with a minimum of long-diitanct 
telephone calls and conferences, #90-3 Average aeani that the 
documentation exists in some form but It Is moderataly hard to operate 
without at least soma discussions with the originators of the doeuaent, 
#90 -6 Uneven/highly variable is put In to characterlia simulation In 
which there is spotty documentation, often indicating an evolution of 
different programmers and different groups working on the model. To 
get decent documentation one may have to search among the diaarray of 
documants that are pregentad. 



DOCEXT, 4/3S 



90-0 


DOCUMENTATION, EXTENT: 


NA; ZERO; UNKNOWN 


0 

■ < < < Ml 


. 3 


90-1 


MCUtffiNTATION, EXTENT: 


EXCELLENT . . .' 


lb. 




90-2 


DOCUMENTATION, EXTENT: 


VERY GOOD 


, , . , 26. 


.8 


90-3 


DOCUJffiNTATZON, EXTENT: 


AVERAGE 


.... J^?. 




90-4 


DOCUlffiNTATlON, EXTENTS 


WEAK 


.... 8. 


is 


90-5 


DOCUJffiNTATION, EXTENT: 


POOR 


1. 


b 

s 


90-6 


DOCUMENT ATI MJ, EXTENT-. 


UNEVEN/HIGHLY VARIABLE 


■ a * • 1*^ • 


90-7 


DOCUlffiNTATlON, EXTENT: 


UNAVAILABLE 


* , * , Pi 






Other CombinationB , , , , 




0. 


8 




m 






i^ 



#91. Documentation Availability/Location: 91-5. Proprietary (classified)/ 
write author. What we mean here Is that the information on the simulation 
ii classified in the Bensa of top secret, secret, and so on. To obtain 
this information, it is necessary to write the author. This relieves the 
burden of identifying the document and approving of its transmittal to 
the author and to the people who are searching for the document. Problems 
of clearance, need to know, and so on can then be resolved between the two 
interested parties. 



DOCLOCl, 4/36-38 

91-0 DOCUJENTATIOJ, AVAIL. /LOC. 

91-1 DOCUMENTATION, AVAIL. /LOC. 

91-2 DOCUtffiNTATION, AVAIL. /LOG. 

91-3 DOCUMENTATION, AVAIL. /LOG. 

91-4 DOCUtfflNTATION, AVAIL. /LOG. 

91-5 DOCUJffiNT^lON, AVAIL. /LOC. 

91-6 DOCUMENTATION, AVAIL. /LOC. 

91-7 DOCmffiNTATION, AVAIL. /LOG. 
91-? DOCUtffiNTATlON, AVAIL. /LOG. ■ 

NR 



NA; UNKNOWN , 

OUT OF PRINT/UNKNOWN 

PROPRIETARY/NOT FOR PROFITS 

PROPRIETARY/GO>MRCIAL 

PROPRIETARY /WRITE AUTHOR . . . 
PROPRIETARY (CLASSIFIED)/ 

WRITE AUTHOR 

PUBLIC/DEFENSE DOCUMENTATION 

CENTER , 

PUBLIC /LIBRARY OF CONGRESS , 
PUBLIG/PROFESSIONAL 

JOURNALS, BOOKS 



6,1 
4.6 
0.0 
10. G 

16.7 

47.0 



0.8 
4.5 



92-0 DOCUMENTATION: GENERAL OVERVIEW (Describe) 

#93. Piilication Type: #93-3. Reports/analyses, etc. This refers to 
publlcationi , possibly generated after a series of runs, to be used as 
an official document, ai a report to a higher authority, or possibly as 
even a aupporting argument for a request for funds. This is in distinc- 
tion to a document which ia a book or article for nonspecific purposes. 

DOCPUB, 4/Z9 

BOOKS OR ARTICLES 

USER MANUALS 

PROGRAM DECKS /LISTINGS 

REPORTS /ANALYSES, ETC 



93-0 
93-1 
93-2 
93-3 



PUBLICATION TYPE: 
PUBLICATION TYPE: 
PUBLICATION TYPE: 
PUBLICATION TYPE: 

93-0 and 93-1 

93-1 and 93-2 

OthBT oombinations 
NJR 



•••••• 



* * * u * 



■ ■ ■ ■ ■ 



1.5 
13.6 
3.3 
24,2 
1,5 
20,5 
25.9 
9.1 
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94-*0 PUBLICATION OR DOCUMENT IDENTIFICATION i 

[Specify one or two documents most ralevant to this game simulation 
or study* Give full references so that documentation may be assembled,] 



TECHNICAL COORDINATION AND STANDARDS 



//95, Technical Coordination: One might have a central clearing house 
which performs a clerical operation with no professional or evaluative 
role. One might otherwise have a staff of several permanant professionals 
whose task Is to compose and to technically describe the inventory of 
modals, aimulations or games. In your opinion, supposing that a central 
clearing house exists ^ ihould it have a technical staff? Discuss, 



TECH, 4/40 



95- 


■0 


TECHNICAL 


COORDINATION: 


HIGHLY UNDESIRABLE 




.. 12. S 


95- 


■1 


TECHNICAL 


COORDINATION' 


UNDESIRABLE 




.. 13,6 


95- 


2 


TECHNICAL 


COORDINATION : 








95- 


3 


TECHNICAL 


COORDINATION ■ 


DESIRABLE 




.. 22.7 


95- 


4 


TECHNICAL 


COORDINATION: 


HIGHLY DESIRABLE 




.. 22.0 






NR 








.. 7.6 



#96. Standards Comnilttee: Question 58 asked about the dasirability 
of standardization without specifying what. Is it premature to try 
to form a professional standards comiaittee for models , gameaj and 
simulations. Is it needed? Would It probably do good or harm? 
Pleaae Conment. 



STmD2, 4/41 

HIGHLY USEFUL 0,0 

USEFUL 14.4 

SAME 9.1 

HARMFUL 41.7 

HIGHLY HARMFUL , ZO.B 

m 14.4 

#97. Questionnaire Evaluation [Written connnents are also welcome.] 
QUEVAL, 4/42 

97-0 RESPONDENT'S EVALUATION OF QUESTIONNAIRE EXCELLENT.,.. 2.3 

97-1 RESPONDENT'S EVALUATION OP qUESTIONNAIRI GOOD 20.6 

97-2 RESPONDENT'S EVALUATION OF QUESTIONNAIRE: MODAL 25.8 

97-3 RESPONDENT'S EVALUATION OF QUEST! AIM : POOR 24. 2 

97-4 RESPONDENT'S EVALUATION OF QUESTIONNAIRE: BAD 12.9 

NR 14.4 

Researohers' evatuation of ths quality of the questionnaire responsea: 

IQC, 4/43 . - 

EXCELLENT 2.3 

GOOD 19.7 

MODAL SB.e 

POOR SO. 3 

Q BAD ^5,2 

ERIC 
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Part III: Man-Machine Games or Simulations and Manual Ganies 



ERIC 
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CHARACTERI ZATION I GENERAL DESCRIPTION 



#98, Control Team: By the phrase "control team*' we mean a formal team 
as part of the game making up rules or interacting with the other teams 
as the game progresses (//98-3), This should be contrasCed with game 
manaiement control (Jt9B-2) ^ where the directors or the managers or referees 
do not play a direct ^ important ^ game-influencing role* For example ^ 
most business games under this categorization do not have a control 
team. Few two-sided dueling games have control teams. Almost all 
poli tical--militaiy exercises have control teafns , 

98-^0 CONTROL TEAM: UNKNOWN 
98-1 CONTROL TEAM: NO 

98-2 CONTROL TEAM:YES, BUT COULD BE COi^UrERIZED 
98-^3 CONTROL TEAMiYES, MUST BE USED 

Low Middle High 

I ^_L^ i J 

Confidence Level 



#99. Number of Live Player Teams* This excludes a control team. If 
the giime has been designed to have a variable number of teams, this 
should be noted. The number of live teams actually used in different 
runs should also be indicated, 

99-0 NUMBER OF hlW PLAYER TEAMS: ( ) Specify 



//lOO, Number of Robots* The same observations hold for robots or 
completely computerlEed teams. In simulations where a coniat system is 
being simulated , such as in a totally computerized duel, we may regard 
the model as having two robots, one for each side playing the other* 
A simple test for this classification would be to ask how to convert 
this game into a man-machine game. In order to do so^ some of the 
automated decisionmaking of one or both sides would be removed and 
replaced by live player decisions. 



100-0 NUMBER OF ROBOT TEAMS: ( 



) Specify 
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f/IOl. Sequencing of Moves: Th^re are some games (such as many of the 
war games) piayed where moves are simultaneous. Furthermore, many games, 
such as two-persnn matrix experiments, usually utilize simultaneous moves. 
There are other games in wliich the moves are in fixed sequential order; 
examples of such are chess or checkers. There are other games in which 
the moves are in variable order; frequently either chance will determine 
tlie next move or a player is in a position to give the move to another 
player, Craps is an example of just such a game; depending on how one 
defines diess, one pauses to see who selects sides at the beginning in the 
first move in an invariable order, after which it is in fixed sequential 
order. Another set of examples is sporting events. In baseball the 
batting order is fixed. In football, the interteam play goes in no 
particular order al though a series of downs is in fixed format. 

In some games some of the moves may be simultaneous, whereas others may 
be in variable order. For example ^ in some strategic war games ic may 
be required to pay costs for force maintenance every period, However j when 
new weapons sytems investment considerations are included^ It is up to 
the individual team to decide whether or not they intend to invest. 

101-^0 SEQUENCING OF MOVES: raKNOra 
lOl-I SEQUENCING OF MOVES: SIMULTANEOUS 
101-2 SEQUENCING OF MOVES: FIXED SEQUENTIAL 
101^3 SEQUENCING OF MOVES ^ VARIABLE ORDER 

Low Middle High 
L - i 

Confidence Level 



#.J02. Moves per Teami In this case, for complex games, we are referring 
to a move by the team as a whole, not to the many individual small transac- 
tions that might be taking place, 

Therfe are some games, such as damage'-exchange rate and attrition e.?alua- 
tion games or continuous search games, in which the simulation or thfe 
computation ii basically a mathematical procedure with no clear 
definition of move. In this instance we classify the move description 
102-0, 

102^0 MOVES PER TEAM: NA;UNKNOra 

102-1 MOVES PER TEAM: MOVES PER TEAM ( ) Specify 



//103* Complenientary Procidufes" 
includes machematlcal solutions 
f/103"6 Includes fie Id exercises, 

103-^0 C0_^1PLE^ffiNTARY PROCEDURES: 

103-1 COhffLEJ^NTARY PROCEDURES; 

103-^ COOTLEMENTARY PROCEDURES; 

103-3 CO^LENffiNTARY PROCEDURES; 

103-4 COMPLEffiNTARY PROCEDURES: 

loa-'S COhff I EMENTARY PROCEDURES: 

lOa^-e COOTLEhffiNTARY PROCEDURES: 



//103--2 includes lectures; //103-4 
ucli as game- theoretic solutlona; and 

NA;UNKNOU^' 
NONE 

LECTURES 

CASE STUDIES /HISTORY 
SIMULATION/ANALYSIS 
"EffERIENCE" 
FIELD EffiRCISE 

Low Middle High 

Confidence Level 



104^ 


-0 


FORMAL 


GA^E 


TYPE: 


NA; UNKNOWN 


104- 


-1 


FORMAL 


GA>E 


TYPE- 


"UNDEFINED, PAYOFFS NOT SPECIFIED 


104^ 


-2 


FORMAL 


Gkm 


TYPEi 


CONSTAT SUM GA>1E 


104^ 


-3 


FORMAL 


GAME 


TYPE: 


NON-CONSTANT SUM GAME 


104- 


-4 


FORMAL 


GAME 


TYPE: 


ONE-PERSON MODEL (mXlMIZATION) 



Low Middle High 

I — t i a 

Confidence Level 



( 
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//105* Rules: //105-1 Rigid manual. An example of a rigid manual game 
would be chess. All of the rules are well spe ified in advance and 
the game is a manual game, #105-2 Semi-rigid manual. An example would 
be a war gamej where, although the fire power and other planning fac- 
tors are supplied ^ some questions during the game may be addressed 
to a referring board and certain rules or rulings are made during 
the course of play. f;105-3 Free-form or referee*s dlreccion would be 
a game such as a political military eKercise in which the control teams 
and the referee -s direction are critical to conducting the game. The 
melding of the moves and adjudication of attempted moves is a critical 
feature of such a game, //105-4 Rigid computerized rules are dis- 
tinguished from rigid manual inasmuch as in general the rules of manual 
games are much more visible and hence much more open to questioning 
than are those of computerized games. One of the major dangers of 
using computerized games is that a great amount of bad modeling and 
theorizing can be hidden in computer programs, //105-5 Semi-rigid 
rules computerized. In this case the game may be computerized , but 
not all of the rules are necessarily described. For example, one 
might have a business or marketing game in which although vircually 
everything to do with sales, manufacturing, and internal running of 
the firm has been computerized ^ press releases and newspapers are 
issued to the firm, thus adding a verbal and somewhat less formalized 
component to the game, #105-6 In some instances games or simulations 
are nothing more than the dynamics of the behavior of a formal mathe- 
matical model or computer program. The category also includes rules 
that are well defined in a game^ theoretic sense* 

In some games, although all rules are given they are so complicated that 
no single individual will know all of them (//105-7) , In other games 
part of the purpose is to discover unstated rules (//105-^8), In free- 
form games, there may be considerable uncertainty eoncerning basic 
structure (#105-9). 

105-0 RULES: NA;UNKNOWN 

105-1 RULES: RIGID MNUAL 

105-2 RULES: SEffl-RIGID MANUAL 

105-3 RULES: FREE FORM/REFEREE *S DIRECTION 

105=4 RULES: RIGID COOTUTIRIZED 

105-5 RULES: SEMI-RIGID COOTUTERIZED 

105-6 RULES: PROGRAM OR FORMAL MATH MODEL 

105-7 RULES: WELL DEFINED BUT TOO LARGE FOR COMPREHENSION 

105=8 RULES: SOM RULES, NOT KNOWN 

105-9 RULES : UNCERTAINTY CONCERNING BASIC STRUCTURE 

Low Middle High 

I- I i 



Confidence Level 
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//106, Structure of Gamei In the category atructure of garaes we include, 
under //106-3 matriK gmm, a game which could conceivably approximated 
by a matrlK even though in some cases the strategies may be continuous. 
For eKample, it may be permitted to have a player pick any pries he 
wants in the range from $0 - $100, but in general he may be limited to 
bids in units of a dollar. Even if he wantad to make it a continuoua 
game, it is quite possible that the machine would round it off by a 
finite approximation and thereby make It a matrix gamst 

Some games may fit into more than one category. For example, a business 
game may have a formal structure such as an iterated matrix game; however, 
it begins play with a scenario describing the state of the iMrket, 
//106»2 Explicit mathematical 2x2 matrix games. //106-3 Other matrix 
game. Here wa are referring to a purely abstract mathematical itructure 
provided as the venue for the game, //106-4 Implicit computerized damage-^ 
exchange calculation which in fact can be regarded as a computation on 
an enormously large matrix. //106-S Iterated matrix game. In many 
instances, such as many plays with the Prisoner's Dilema, the game is 
played in a dynamic mode. However , a great amount of the decisionmaking 
is performed on the same scructure pariod after period. Many buslneai 
games have the aama "battlefield" of a more or leas similar market each 
period. //106-6 Free form extensive . Games like political military 
exercises would fall in this category. They are played move by move, 
but they are not necessarily repeating the same situation on each move. 
#106--7 Formal eKtensive* A game such as chess is played in formal 
extensive manner. The rules are rigid and well-defined; however, the 
players move play by play and do not enunciate overall strategies for 
this game. The remaining two categories concern games in characteristic 
function form or games in whidi the prime area of inveetigatlon ia coalitions 
//106-8 Free form characteristic function* Such a game would be one 
devoted to studying the coalition possibilities for a treaty on the use 
of the Danube, as just one example. //106-^9 Foriml characteristics 
function form* There have been experiments done using games apeclfled 
in characteristic function form where the eKperimenter studies how the 
players divide the money they receive from cooperative acts. 

Question #106 calls for some familiarity with several concepti of game 
theory. If you are not sufficiently familiar with the terms to answer 
check #106"11 and proceed to the next question. ' 
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i^l07. Information State* //107-1 and llie first refers to perfect 

information in the game-theoretic sense: all players know all that can 
be known at all times. An example of a game with perfeet information 
i« a chess game. Incomplete information is the situation that prevails 
in a poker game* The kibitzers or a referee may know what the hands of 
all the players look like, but the players do not know each other *i hands. 
i^l07^5 Considerable structural uncertainty refers to games in which the 
rules and the general environment are by no means clear at the beginning 
of the game. An example would be an extremely free form political garner 
to be played in a future time period with the skitnplest of scenarios 
available, ifl07^3^ 1H07-6 In some games that are designed to teach 
procedures and approachfis to a pfoblem, it is worth distinguishing 
Information states in which information can be bought as contrasted with 
those in which calculation can be bought. For example , in a business 
game one may have information concerning the reaction of the market to 
various arrays of prices. On the other hand, one may not have procedures 
for fitting modals to this information. The procedures such as least^ 
square statistical packages can be regarded as calculation packages. In 
some cases these may be available to players from *' consultants'' who charge 
for their use. 



107^0 
107-1 
107-2 
107^3 
107-4 
107-5 
107-6 



#108. 



INFORMATION STATE: 
INFORMATION STATE; 
INFORMATION STATE: 
INFORMATION STATE: 
INFORMATION STATE? 
INFORMATION STATE? 
INFORMATION STATE j 



NA; UNKNOWN 

PERFECT 

INCOOTLETE 

INFORMATION CAN BE BOUGHT 
SOME RULES NOT KNOWN /MIXED 
CONSIDERABLE STRUCTURAL UNCERTAINTY 
CALCULATION CAN BE BOUGHT 

Low Middle 



High 
1 



^onf i Jence Level 
Under //108-1 bookkeeping/light staff workj we Include 



Computer Use 

the mm of the computer for somewhat more than straight bookkeeping, but 
nomawhat less than one might wish to describe as hea%o^ analysis* #108'-6 
Man^machine on-line interrogative mode, We distinguish this from man- 
machine interactive in the sense that interactive merely implies that the 
tnachine does the computations on the environment, whereas interrogative 
Implies that in the process of calculating^ the irqchine questions the 
player and obtains answers from the player. 
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^a09 Gaming Facilities: f/l09-7 This refers to the situation where a 
special system set of languages oi program may have been written to 
accompany the running and general handling of a specific game. For 
example 5 some games run at the labs at Berkeley and some games run with 
the TRACE system at SDC or at UCLA depend upon the availability of 
much of the specialiEed extra hardware and ioftware. Although It 
is possible that the games themselvei can travel, much of their power 
is lost when the accompanying programs and equipment are not available, 
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//111, Elapsed time of run^ start to finish: We refer to the playing time 
of a single game ors in the case of experimental games, of a single experi- 
ment which could involve several replications j for example, when a aeriei 
of experiments is run sequentially over several dayi * In som Instances 
the nature of the game and its format make this figure quite precise. 
In other cases there is a variability of several orders of magnitude 
concerning how long the game takes to play, For eKample, some buiinesg 
games are run on the basis of one decision a day or one daclaion a week 
that is made in less than an hour* otheTOise, the individuals carry 
on their normal routines. 

Under this category we refer to total elapsed time from the start of the 
game to the debriefing, In some Informal instances, such as running a 
game with a class, one may run the game for the whole of a semester, For 
example, the Carnegie Tech game may run for a period of seven or eight 
months. The games of the Studies, Analysis and Gaming Agency are frequently 
run in two different modes; One is a 3-1/2 day intensive game, and the 
other stretches over several weeks. In the case of eKperimental games, 
games are often run in parallel — possibly intensively during one evening 
for the whole of a game, but in some cases batches of games are run □'mpri- 
sing an experiment. In other cases playera may run for more than one day. 
If a game is run on more than one mode, the different elapsed times should 
be indicated. 



111-0 ELAPSED Tim OF RUN, START TO FINISH I 

Low 



Middle High 
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//li2* Player Game Play Time' By this we mean the amount of time spent 
in actually playing a game. This includes briefing, decisionmaking, and 
debriefing associated with the game. 
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PLAYER 
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PLAY 
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#113, Formal Game Prebrlefing Procedure i We note that the military uses 
the word "indoctrination" when describing materials sent out prior to 
the formal briefing time in a gaine. 

The dlitlnctlon to be made here concerns whether or not a game has a 
formal prebrieflng procedure or whether the prebrleflng iis informnl or 
nonexistent. For example , chegs players in general need no prebrleflng 
if they already know the rules. Some simple games, auch as experiments 
with 2x2 matrix games j may be run with an informal briefing from Che 
experimenter who has either decided to dispense with formal contrcl or 
has overlooked the use of formal control In the verbal description of 
the game. 

113-0 FORMAL GAJffi PREBRIEFING PROCEDURE' NA;UNKNOra 
113-1 FORMAL GAME PREBRIEFING PROCEDURE : NO 
113-2 FOH^ GAm PREBRIEFING PROCEDURE: YES 

L . Middle High 

L . I J j 

Confidence Level 
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#114, Formal Briefing TiTna (%) i This can be described as a percentage 
of player ganie^play time. For example, if it takes a player ten hours 
to play a game and there is a briefing session of 1/2 an hour, this 
means that briefing time is five percent of game play time. 

114^0 FORML BRIEFING TDE (%) i ( ) [Specify] 

Low Middle High 

Confidence Level 



#115, Debriefing Time: Thii is also stated as a percentage of game 
play time for the player. In general, many experimental games, and 
certainly games for antercainment , have little if any debriefing. 
Occasionally there are post mortems after chess. Operational gamts and 
games for teaching and training may have considerable debriefing. For 
operational use, the length of debriefing is fairly clsaf; for example^ 
after a SAGA game, there may be a half day set aside (beyond the three 
days of play) specifically for discussion and formal debriefing* How^ 
ever J with a game used for teaching purposes, such as the Carnegie Tech 
gamej one might regard the complete course taken with the game as a 
brief ing^debrlefing session; In that case, one could claim that the 
debriefing and briefing time could easily be as large if not larger than 
the game-playing time. 

If a game is used for different purposes one should indicate the briefing 
and debriefing time of each. The context of purposes stated in the earli 
question should make clear the specific category to which a game belongs* 

115^0 DEBRIEFING TM (Ji) i >15 ( ) [Specify] 



Low Middle High 
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#116. Control Time, Total Expenditure: This includes briefing time, 
running time, and debriefing time of the control group* For example, 
in a SAGA exercise, there may be field trips and so forth before the 
gamers scenario can be written, This type of work would be classified 
under game corstruction* The time we are Interested in here is that 
amount spent by the individuals GOmposing the control team for running 
purposes* If it is necessary for meinbers of the control team to be 
briefed or Indoctrinated for several weeks in advance, this would be 
counted as part of control timei If, however, as is usually the case^ 
they Join the group merely a day or two or even less before play time, 
we would start to count control team time from this point. In some 
instances there is not a formal control team; however, there is never^ 
theless a game director — formal or Informal — whose time is being 
used to supervise the process. This should also be counted even though 
this will generally amount to no more than a few hours or a few days. 

116-0 CONTROL TIME, TOTAL EXPENDITURE i MAN-^WEEKS ( ) 

[Specify] 

Low Middle High 

■ - -t J 

Confidence Level 



/^117. Post Debriefing Analysis (Intensity): This refers to the 
analysis of the game run after the game is over; the debriefing may 
have taken place. In other words, this should not be confused with 
debriefing • It refers to the analysis which may be done by researchers, 
possibly the players in a different mode, or others to determine what 
has been learned from the game. In the case of exparinients , this is 
quite obviously where inuch of the work is concentrated. In the case of 
operational games ^ this is where much of the work should be concentrated 
if one wishes to measure the effectiveness of the exercise* #117-0 
Post debriefing analysis, For straight simulations there is no debrief-- 
Ingj hence this category is not applicable. At the same time simulations 
invariably involve analysis after they are runi this is picked up in 
#118, itin-U^ Considerable, would apply where the analysis time may be 
even more than the game-playing time* 



117-^0 POST DEBRXEFING 

117-1 POST DEBRIEFING 

117-2 POST DEBRIEFING 

117»3 POST DEBRIEFING' 

117''4 POST DEBRIEFING 



ANALYSIS (INTENSITY) 

ANALYSIS (INTENSITY) 

ANALYSIS (INTENSITY) 

ANALYSIS (INTENSITY) 

ANALYSIS (INTENSITY) 



: NA; UNKNOWN 

: NONE 

I SLIGHT 

I MODERATE 

• CONSIDEMBLE 

Low Middle High 
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#118. Analysis Time" Answer this question In terms of man^weeks, or 
the percentage of actual time spent by individuals involved in the 
analysis as compared with total game play time* There may be an enormous 
amount of automated analysis going on with a small eKpenditure of hutnan 
time* This question is concerned with the human timCi 

118-0 ANALYSIS TIME: ( ) [Specify ] 



119^0 SET UP COSTS AS % OF TOTAL COSTS TO OPERATE : (_ ) 

[Specify] 

Low Middle High 



Confidence Level 



#120t Cost to Operate at New Location: EKcept for going to the new 
location we assume that costs will be distributed the same way. In 
other words, if one is calculating on free secretarial help at one 
place J one calculates some free secretarial help at the other place. 

It makes a great difference whether or not you can bring In an operating 
crew. However this means that cost to operate should be looked at as 
the minimum cost to bring in att operating crew or coat to train new 
people on location* If no crew Is available , it may be either impos- 
sible to transfer the game or inordinately expensive. 

When we refer to new location we assume that the new location has hard- 
ware tLiat is suited for the game involved t 



120-0 COST TO OPERATE AT NEW LOCATION i ( 



) [Specify] 
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GmE CHARACTERIZATION i PLAYERS 



//121, Player Selection. //121-5 "Proprietary interest advoGatas*' refers 
to the use of playeri who have a personal interest in the actual use 
of the game or in the use of the results of the game for some specific 
purpose. ^ For sKample, a proprietary interest advocate might be a group 
of individuals either advocating or opposing a weapons system, such 
as MIRV, or the SST^ or a specific piece of hardware. The game may 
be used as part of an ongoing advocacy process. In cases such as 
this J it is extremely important to sort out players whose play sur- 
rounds the environment of the game from players whose interests cease 
with the use of the game for whatever its explicit, stated purposes. 
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//122, Player Characterizatloni Postgraduate refers to Individuals at 
a war college or other academic Institution. #122=3 "Professional** ap- 
plies to the context of the game being played; e.g,, a military man 
playing a war game would be regarded as a professional. If he were 
playing a business game, he would not, in general, be regarded as a 
professional * 

122»0 PLAYER CHAMCTERIZATION: UNKNOWN 

122-1 PLAYER CHARACTERIZATION i POSTGMDUATE 

122=2 PLAYER CHARACTERIZATION ^ ADULT (NONPROFESSIONAL) 

122=3 PLAYER CHARACTERIZATIONS PROFESSIONAL 

122-4 PLAYER CHAMCTERIZATION I OTHER 



Low Middii High 
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f?123. Player Use, In some man-machine exercises live playeti are used 
only because they are cheaper or more readily available than a almu- 
laced player. There is no attempt to train them nor are their goals of 
particular concern to the exercise , In this sense they are merely a 
substitute for machinery; this posiibility is described in #123-0, 

123-0 PLAYER USE I ONLY AS ''MAGHINERY'* 

123-1 PLAYER USE: AS PLAYERS 

[If answer to #123 is //123-'0p skip the remaining questions on players 
and go to #135, ] 



124=0 PLAYER 

124-1 PLAYER 

124-2 PLAYER 

124-3 PLAYER 

124-4 PLAYER 

124-5 PLAYER 

124-6 PLAYER 



PAYOFFS : UNKNOWN 

PAYOFFS J MONEY WAGE 

PAYOFFS; GRADES OR PAYMENT 

PAYOFFS i FIXED PRIZE 

PAYOFFS' PRIZE PROPORTIONAL TO PERFOW^CE' 

PAYOFFS I "EDUCATION*' 

PAYOFFS I NOT SPECIFIED 



#125. Player Preteit Comprehension Testi Is a cheek made to sae if 
the players fully corapfehend the game prior to play? This includes 
the use of a practice play followed by questions, 

125=0 PLAYER PRETEST COICPREHENSION TESTi UNKNOOTJ 
125-1 PLAYER PRETEST COJffREHENSION TEST! NO 
125^2 PLAYER PRETEST COHPREHENSION TESTi YES 

Low Middle High 



i — , . ' > 

Confidence Level 



//126, Player Pretest i ^^126-1 refers to the case where no particular 
pretesting after the selection of the players has been performed; 
#126-2 refers to games in which one has in fact run pretests on the 
players. This may be a California Personality Inventory, it may be 
IQ-teatSj and so forth. One runs subjects or players through a bar- 
rage of tests outside of the formal garaep 

Player pretest could be a test for IQ, general knowledge, etc., or 
some sort of personality test* It does not necessarily imply a compre 
henslon test for the game, 

126-0 PLAYER PRETEST I UNSOWN 

126-1 PLAYER PRETEST i NO 

126-2 PLAYER PRETEST: YES (Describe) 

Low Middle High 

i — ' i 

Confidence Level 



127-0 PLAYER POST PLAY COMPREHENSION 
127-1 PLAYER POST PLAY COOTREHENSION 
127-2 PLAYER POST PLAY COtffRIHENSION 



CHECK I NA I UNKNOWN 
CHECKS NO 
CHECK: YES 

Low Middle High 
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#128* Player Perception of Success of Purpose i In this charaeterl^ 
zation we are not asking the question whether the game was effective, 
but how the experience was perceived by the players. A good question 
that must be asked of all games iSj ''How does enjoyment correlate 
with the value of a game?'^ It is conjectured by us that up to a 
certain level there is probably a positive correlation between the 
effectiveness of a game and the level of enjoyment or enthusiasmi 
Beyond a certain level, however, we suspect that the correlation weake; 
or goes negative. A highly enjoyable game may in fact hava been 
enjoyed as a game and not as an operational ^ research, or teaching 
device Informally we have observed that apparently there is not a 
great amount of correlation between highly popular lecturing , acting 
performances, and the amount of information that is conveyed to the 
students. In some cases player reaction may be mixed, You may wish 
to give a percentage breakdown next to the categories or draw a small 
graph. 



128-0 PLAYER PERCEPTION 

128-1 PLAYER PERCEPTION 

128^2 PLAYER PERCEPTION 

128^3 PLAYER PERCEPTION 

128-4 PLAYER PERCEPTION 

128-S PLAYER PERCEPTION 



OF SUCCESS OF PURPOSE: 
OF SUCCESS OF PURPOSE: 
OF SUCCESS OF PURPOSE: 
OF SUCCESS OF PURPOSE i 
OF SUCCESS OF PUWOSEi 
OF SUCCESS OF PURPOSE i 



NOT RELEVANT; UNKNOWN 

HIGHLY POSITIVE 

POSITIVE 

NEUTRAL 

NEGATIVE 

HIGHLY NEGATIVE 

Low Middle High 
Confidence Level 
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#130, Number of Players Per Teami Do not fill in more than the three 
most frequently used categories* 

130-0 NWfflER OF PLAYERS PER TEAM: (Specify) 

Low Middle High 
Confidence Level 



131=0 REPEATED USE .OF PLAYERS: NA^UNKNOWN 
131=1 REPEATED USE OF PLAYERS: NO 

131=2 REPEATED USE OF PLAYERS: YES ^ . 

Low Mladle High 

■ ? 1 

CDnfidence Level 



//132, Role Playing (Individual): #132=1 Role playing: self is the 
category which indicates that no other role playing la required of the 
players in the particular game* //132--3 Role playing! specific person 
implies for example that somebody play Stalin or Mao Tse^tung or 
Mr* Nixon* //132-4 and //132--5 A specific organization might be something 
like General Electric; an abstract organization would be a large husineas 



firm* 
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#134 . Importance of Unstated Purposes i This question InVDlves the 
characterization of players and what might be described as "the game 
outside of the |^ame." For example, in the case of a business game 
where many exec^itives from the same firm play unaccustomed roles in a 
simulated hierarchy, there may be pressures exerted on the Individuals 
as a result of their being aware that they are being watched. Even in 
experimental games , the players sometimes Inay decide to play "fool the 
experimenter" or "give him what he wants*" This question is admittedly 
subjective, but it merits serious consideration* 



134-0 IMPORTANCE OF UNSTATED PURPOSES 

134-1 IMPORTANCE OF ^STATED PURPOSES : 

134-2 IMPORT^GE OF UNSTATED PURPOSES: 

134-3 imORTANCE OF UNSTATED PURPOSES: 

134-4 IMPORTANCE OF UNSTATED PURPOSES: 

Comments or Discussion* 



NA 
HIGH 

' SOMEWHAT 
SLIGHT 
NONE 



Low 



Middle 



High 



Confidence Level 



#135, Game Users; Count the number of institutions where a game, sim- 
ulation, or a direct variant is being used, This may tend to produce 
some overeatimatlon^ yet for most purpoies this is the most relevant 
figure. It gives an insight into how widaspread the direct use of a 
game has been or how widespread the Influence of a game has been. 

135^0 GA^ffi USERS f (Specify). 



Low Middle High 
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Part IV: Voluntary ABsessment 



ERIC 



-137= 



■ontpletion of thin sffation^ 
V'jUS, 4/44 

i'i I not ao^npl^ 



For the remaining questions assuine chat a senior professional 
(one who really knows tlie business) wishes to usa this MSG and 
wanes your evaluatioi* along several dimensions^ 

f;^36. Assessment = Design and Cons c rue tioii : Wliat are the strengths 
and weaknesses of this MSG - s design and cons true t Ion? 

Comment! 



#137, Assessment - Planning Factors and Data^ What In your opinion 
are the strengths , weaknesses , and constraining features of the data 
used in the MSG? How serious are the deficiencies or weaknesses, 
if any? 

Comment i 
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#138, Assessment - Documentations Hew complete and Meful is the 
supporting documentation? Would it be easy for some other agency 
to use the MSG, or would the extent and quality of documentation 
mke this difficult or Impossible? 

Coninent- 



#139* Assessment = Operation : Are there pecullarltieE of operation 
that a prospective user should be aware of? Is the MSG easy to operat 
or are there unique proeedural problem that one should know about? 
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[For Man-Machine or Manual Garoi Only,] 

#140. Asieiiment - Post Debriefing Analysis! ktm thm MSG'i outputi 
easy to analyze or are thty Intended for mm in ■ubaequMt analyses? 

Cooments 



//141, Aisesiment - Coit Effectiveness i Do you think that the MSG 
represents a cost-eff active way to get at the iisues it addresses, or 
would you recommend alternative procedures, methods, or techniquas? 



Conmeij c i 



iH^I, Assessment - Validation Criceriii: V/hat questions related to 
validation have been posed ^ and are they clear and concise or are thev 
vague, confusing or non-txistent? Has much attention been given to 
validation of the MSG? 

Comments 



//143* Assessment ^ Validation 
developed^ was any validation 

Comment * 



Based on the criteria that were 
done on the MSG? What resulted? 



#144. ABsessment - Overall: In your opinion is the MSG of 

outstanding J average ^ poorj or of indeterminate quality? Would you 

commend It for future use? Unqualified acceptance? Qualified? 

Comments 



Appendix B 
MODELS, SIMULATIQNS, AND GAMES SURVEYED 

FIRST VERSION OF QUESTIONNAIRE ^ 
mAY 

Inter-Na tlon S imula tion 

Business Game for Teaching and Research Purposes 
SECOND VERSION OF QUESTIONNAIRE 

QUICK (Quick Reacting General War Gaming System) 
GO 

FAST-VAL 

Division Battle Model 
CAIR=-COhff 

SIDAC (Single Integrated Damage Analysis Capability) 

Supply Point Simulation Model 

SIGM^OG Theater Materiel Model 

TCM (Theater Combat Force Requirements Modal) 

(Laboratory Problem 2)^-'ICBM Logistics Simulation = 
CARMONETTE 
CASOADE III 
SATAW 

FINAL VERSION OF QUESTIONNAIRE 

Time Reported 
To Complete 
Quest ionnalre (hr) 



1. LEGM. tax IV 1 

2. EFFECT 3&4..., 5 

3. SMILEY 13 

4. NOMOS=NOMOGR^HS 12 

5. VALUAT V 14 

6. Cost Effectiveness Model * * i # t » • , • . 6 

7. Evacuation Policy Model 3,5 
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Time 

8. TAM (Target Acquisition Model) 8 

9. DYNTACS X— Small-Unit Combat Simulation 2.5 

10. HOVARM 7 

11. New Unit Cost Model, Modified 4 

12. DYNTACS (Dynamic Tactical Simulation) 3.5 

13. STANO— Systein Assessment Model, Phase 1 (SAM I) 8 

14. Trans-Hydro Craft 7 

15. Fire Support Simulation 5 

16. APSUB MOD 0-1 1^5 

17. CPOAM/TACOPS (Tactical Operations Model of Continuous 

Fleet Operations) 7 

18. APAIR 8 

19. APSURF 2.5 

20. LOTRAK 1I~ASW Localization Model 5 

21. ASGRAM 5 

22. Nuclear Exchange Model III 5 

23. STEM (Sub Trailing Evaluation Model) 4 

24. POSTURE 1.5 

25. SUBDUEL 1 4 

26. VERS (Vehicle and Equipment Requirements) 8 

27. SASWEM (Sub ASW Engagement Model) I5 

28. Detailed Ship Loading 6 

29. VBffBOLT (Vehicle Mission Processor Based on Least Tiroes) 5 

30. TATAWS 3 

31. FLAME I 13.5 

32. EXO-1 5 

33. FAIR PASS 11.5 

34. LOADER 3 

35. SPARE (Strike Planning, Aircraft Requirements Evaluator) , 5 

36. SIMCAT (Simulation of Contingency Air Terminal Model) 3 

37 1 ATLAS II ,. , , 6 

38 ., FAME 5 

39. ASWAS (ASW Air System Model) 5 

40. ATMIX— Individual Unit Action 4 
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41. CODE 50 (Navy) 3 

42* SAAB (Simulated Air-^to^-Alr Battle) j 

43. mRADS . _ . . _ 9 

44. ASESEM (ASW Escort Engagement Model) , 6.5 

45. SPOL (Shore Parcy Operation and Logistics) ,,,, A 

46. SWEM (Strategic Weapon Exchange Model) 10 

47. CAM-SAM _ , , 3^ 5 

48. Minefield Analysis with Hunting Evaluation Model * . . 5 

49. STS-2 (Strip-to-Shore Model) , 6 

50. SAMEN _ _ , , , ^ 5 

51. ORION . , , ^ . , _ X4 

52. LEGION _ , 20 

53* TARTARUS IV N/COCO J5 

54. COBRA (Comprehensive Blast and Radiation Assessmant 

System) .^1^ 

55. AEM (Arsenal Exchange Model) ...i,.,, 4 

56. SINUS (Small Infantry Unit Simulatorj forinerly 

CARMONETTE) , , ^ 2 

57. ETNAM , 1 

58. SOAR 2 

59. TRACS , . , , , . _ ^ ^ _ I 

60 . TRAMPS , , , _ 1 

61. MORG , . , , 2 

62. MACE ,,,,,,,, , _ 1 

63. THEFT , , , , _ , 1.5 

64 . Localization LASCAR , . , , , , 9 

65 . NETWORK SIMULATOR , . , , 9.5 

66* ASWASP . . , . , , , _ , _ _ _ 8 

67. AAWSEM (Antl^Air Warfare System Effectiveness Model) 6 

68. TACOS II , , , , 32 

69. DMEW (Deterministic Mix Evaluation, Worldwide) 24 

70. GRAPHICS 32 

71. GFE III (Gross Feasibility Estimator) 14 

ERiC VALIMAR , 
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Time 

73, AREA DOMINATION II 4 

74, TAR III (Target Acquisition Routine) 15 

75, FORECAST II , 27 

76, TAFCOM , _ , ^ 4 

77 . SAM , , . _ , 4,5 

78. Air Contingancy Terminal Simulator , 4 

79. GIANT , 8 

80. TACAIR 69 

81. OASIS - 71 , 28 

82. PEGASOS (Penetration Evaluation Gaming Analysis 

Strategic Offensive Studies) 24 

83. Advanced PenetraCion Model , , . . , , 3 

84 . WEAPON , . , . . 24 

85 . TAG AVENGER , , . , , 10 

86. CO^fflO — Combat Bombing Weaponeer ing Program , 4,5 

87. OTASS , , 13 

88* SADDLE . , . , , , , 19^5 

89, ATOM (Air Tactical Operations Model) 4 

90, ENDO-1 . , , , , , , _ ^ 5 5 

91, "Candidate Families** Methodology^ Simulation^ nost, 

and Optimisacion Models * . p , . , , , , , _ _ 4g 

92, ATLAS (Tactical, Logistical, and Air Simulation) 40 

93, Global Distance Routine 10 

94, INFANTRY 69 

95, SPHINX II (Survival Probability Hazard in a Nuclear 

EKchange) , p , 25 

96, SOUCA—Soviet Capabilities 14 

97, NEWCON , , , . . . 80 

98, FASTALS (Force Analysis Simulation of Theater 

Administrative and Logistics Support) 10 

99, SOURCE , 8 

100* SDPS (Space Defense Planning Simulator) 19,5 

101, OFD (Objective Force Design) 20 

102. PFD (3=Preliminary Force Design) 13 



Tinie 

103, DELOGREQ (Deploymenr Logistics Requiremints Model) 4 

104* PFD-SAM (Preliminary Force Designer Intertheater 

Movements Simulation) , ,«..«...*««...... ^ ^ t . , 35 

105 , BOmSIM , 4 

106. UNCLE _ _ _ . 3 

107. STAR-III * . . , 6,5 

108. OPSTRAS NR 

109. CEM (Campaign Execution Model) h 

110. SWIM II 3,5 

111. DIWAG * , 3 

112. APSURV Model 1 2 

113* ADM (Atomic Demolition Munition) . . * . ^ . . , . , , . 14 

114. SEANITEOPS AGGREGATE COST MODEL , 5 

115. THEATERSPIEL , , 8 

116. TBM (Theater Battle Model) 4 

117. TACSPIEL (Computerized) 16 

118. GLOBAL , . , , , , , , 8 

119. ADVICE IT . , , 5 

120. DIVTAG II 3 

121. DBM (Division Battle Model) 6 

122. COMMEL . . , , 40 

123. HOVER , , . , , , 9 

124. TEl^l (TACFIRE Effectiveness Evaluation Model) ., 20 

125. Modif ied^Filter . . , , 10 

126. SUB-AIR BARRIER 4 ' 

127i Strategic Force Mix Model 1....,.,... 5 

128. Corps Battle Model , , , , , 8 

129. SYNTAC \ , , 28 

130, APCAMP (ASW Program Campaign Model) 3,5 

131, STRNm 8 

132. Multi-^Ship ASW Simulation 4.5 

1 33, VIM (Not included In evaluation) 
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Appendix C 

SURVEY CQDEBQQK AND LIST OF VARIABLES 



Cdrd 1: 





V 1 r" 1 ^ b 1 e 
Plumber 




Df finl PQh 










Lard 








■ t « , = 4 J. 




lb 


SenuincG nurier of tnp 


























Te* t 






7B 




Reioondint's Titlg 


Text 









CARD 




integir 


1-2 






H 


5iquinee nunber gf 
ObSirvatign 


Inttgif 


3-5 




SB 




L-eligninq gr hutldlfig 
AgiriCy or f i rn 


Text 


6-30 






AUTHf R 


Author's niitif of <'SC 


Ttxt 


31-50 




SO 


SPUNSn 


Spenser fispo^itbla for 
Inlcidtfon 


Text 


51 -7n 




2E 




Agency oFdering the work 


Text 


7N80 
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Card 3t 



General 


Vari able 
Number 


Vari able 
Nanie 


Defi ni tion 


Attributes 


Column 


Office 




CARD 




Integer 




Office Use 




ID 


Sequence number 


Integer 


3^5 




1 


QT 


Questionnaire Time: hours 
to coniplete questionnaire 


Real number 




D^scri pt i on 


3 


CATEGl 


Des iQnfltion of Item 3S MS G 


0 - Model 

1 ' Simulation 

2 - flan "Ma en me 

3 " Manual 

4 - Analysis or 

Study 

5 - Other 

9 - No responsg 


10- 11 f ?1 




4 


RESROLl 


Respondent (to questionnaire) 
role 


0 - Funder/sDonsor 

1 - User 

2 - Design/build 

3 - Player 

4 - CaretaKe?^ 

5 - Control 

6 - Other 

9 - No response 


12-14 [3] 




5 


RTIHI 


Length of time respondint 
acquainted with work, ex- 
pressed in months 


Real Number 


15-16 




10 


PURPl 


Major stated purpose of the 
work 


0 - Tech. fva; , 

1 - Doctrinal 

Eval. 

' Force Structure 

3 - PDM/IR 

4 - PME/ Domes tic 

5 ^ T/E 

6 - Research/Method 


17^19 [3] 




t 






7 - Other 

9 - No Response 






lOCL 




Confidenci level for PURP 


0 - No Response 

1 - Lm 

2 - Middle 

3 - High 

4 - Absolutely 

certain 


10 



First or dominant choice goes 1n first columni second goes Into second, etc. If only one choice, ranalning columns in 
(J" Id are left blmk. [2] - Two posslbllltlis* 

ERIC 
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Card 3~cDnt Inuecl 



Gf neral 

CI iSS 



DeicriptlDn , 
cont. 



Variable 
Numbif 



12 



13A 



133 



14 



14a 
19 



17 

(17 & 18) 



Vari able 
Name 



CLASWO 



CLASWIl 



PROREV 



REVDAT 



MSGPAR 



MSGPRO 



DEVTMl 



Diflnltion 



Clailif Ication Of MSG 
[without Inputs) 



Classification of MSG 
whin data added 
(highest noted first) 



Has MSG biin given pfoffs- 
sional external review? 



How many monthi iince last 
professlgnil review of 
MSG? 



f^'untser of fiSG parents 
(direct) 



Confidence level for fiSCPAR 
Nuntir of f'SG spinoffs 



Total time elapsed LGtween 
developnient ^nd Inflation 
expressed in months 



Attributes 



0 


- 




1 




FCJUO 


£ 




Conf i d, 


3 




Secret 


'I 




TS 


5 




Prop rie til ry 


£ 
o 




( if h 






(4 1/ 1' tj 3 |..'y f 1 3 e 


0 




f^A; Unci ass , 


i 




r/Ti If I 
1 uuu 


L 




Con f 1 d . 


J 




Secret 


/\ 






i; 




Propriety r y 


f! 

t) 




L! T n P 1 


9 




No Response 


0 




No 


1 




Yes 


q 




fiO Response 


0 




fiot done 


1 




-6 nonths 


Z 




6-12 months 


3 




13-24 months 


4 




25-36 months 


5 




■36 months 


9 




U.o Response 


0 




!^one 


1 




One 


2 




More than one 


9 




No Response 


(See Col. 20) 


0 




iipne 


1 




Cne 


2 




fiore than one 


5 




-No responsf 


0 




•■3 months 


1 




3-6 montHs 


2 




7-12 fnonths 


3 




13-18 nionths 


4 




19-24 months 


5 




25=30 months 


6 




31-36 rT'onths 


7 




37-42 months 


8 




M months 


9 




No RespunSG 



Colunin 



21 



22-23 [2] 



24 



25 



26 



27 
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Card 3^-continued 



General 
Class 



MSG Production ana 
Purpose 



VariaBle 
Numbir 



Viriabl e 
Nimt 



WHOl 



It 



23 



24 



WHODlOl 



INPURPl 



24(:l 
25 



25CL 
26 



SPCPRP 



ALTPR01 



Diflnition 



Source of funds to build KSG 



Who produced flSG? 



WhQ Initiated tht fnodel? 
Who was responsible for 
gilting v/erk unriirway? 



What was tile Initiator's 
(Variable ff23) purpose 



Confidinci level for INPURP 

How spicifie was funding 
source in. dtslgnation of its 
purposi? 



Confldenci 1iVi1 for SPCPRP 

Bist alttrnitivi proeedurti 
to thi MSG 



Attributes 



i 



0 ^ Uni versl ty 
\ - Foundation 

2 - Ppivate 

3 - mh 

4 - JCS 

5 = USA 

6 - uSAr 

1 - USN 

8 = Other COD 

9 - Other U.S. Gov. 

0 - Private 

1 - university 

2 ^ For profit 

3 - Anried Forces 

4 - i^ot for Profit 
y - No neiponii 

0 - fiAtUnknown 

1 ^ Ouilders/risearch 

2 ^ In hyuii/Usir 

3 ^ In house/non-ustr 

4 = E^terna1/uier 

5 ^ External/non^user 
i - flo .lespons^ 

0 - NAUinknown 

1 - Teach/Trng, 

2 - Ana1/Diag. 

3 - Ops. 

4 = Expir. 

5 ^ Research/theory 

6 ^ Advocacy 

7 - Other 

9 ^ No HiSponsi 



0 - NAi Unknown 

1 - Tight 

2 - Moderatt 

3 - Diffuse 

9 - No Response 



0 - NAiUnknown 

1 - Mone 

2 - Licturei 

3 - Ciiii/history 

4 - Arralyiii 

P - Expiritnct 
6 - Gaming 
9 - Ho Response 



Col unln 



30-33 [4] 



34-35 [2] 



36 



37^39 [3] 



40 
41 



42 

43-44 [2] 



ERIC 
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Class 


y i r i a b 1 1 
Numbe r 


Variable 


Defi nition 


Attf ibutes 


C 0 1 gmn 


HSO Pf^Qduetion 
and Purpose , 


27 

27CI 


USE1 


use fi*^ f'ju? [Order tritr 
r-esL/t^t to ^£^4] 

Cnrifidtnce level fyr MSE 




■45^4/ [3] 




3u 


BRIEF 


i!Ul- lju f of LmHo f ^ *h;s biUtnl 
an tne MSG 


r = 4.6 

J = 7-9 . 
a - in=i2 

& = 1 J = i b 

6 - If- in 

7 - 19-i!l or i"are 
ft - LnLnnwin 

9 " No ^eljpDnse 


A'- 


"ij Costs 
to ^'jiid 


31 


CSTl IC 


Direct com to build 


0 - ^ %m 

1 - 50=?9r. 

? - 100-249K 
3 - 2S0-49yf: 

I . 7S0-99^r; 

6 = U2.4g Millian 

7 - £ . 5^5 '■ill ion 

8 r '5 Million 






3SCL 




Confidtnci? level for tSTiiln 


9 - NA', UnJ^nown; 
No Responio 


rn 




35A 


C5TFAr 


Direct colts for all in 
serTfi 

Confidence level for CSTFAf! 


Sipig as **35 






36 


DIRFDS 


Direct funds usid to build? 


0 ^ No 


^4 




37 


FUmUIF 


Dlrtct funding to biHld 


1 - Yii 

5 - iiO Response 
£amu di =35 


55 




3Hl 




Cenfidence levirl for Fu.DIft 








38 

38CL 


TCTCST 


Total gf all types of coit 
to Luiid MSG 

Confidence livel for TUTCST 


Saine aS *35 


&7 

&a 


'•^s* Costi to 
gee ra tt 


3D 




Average annual qp^'raiinf: 
colt 


0 - <3p; 

1 - 10-^4K 

2 - 25-4'jp: 

3 - 

4 = 75^39K 
1 - MOOF 

9 - flQ r.ispnnse 


L9 



-151- 



Card 3 — continued 



General 
CI aiB 


Variabli 
Numbir 


Variable 
Naine 


Dif 1 nition 


Attri butes 


Col unin 


MSG Costs to 
Operatei cont. 


40 


CSTRUN 


Confidence level for CSIOPS 

Operiting cost for a single 
run 


Sanie aF ^3S 


60 
61 




41 


CSTUP 


Annual update costs 


Same as #39 


62 




41CL 




Con f i den ce- 1 e ve 1 CSTUP 




63 




42 


OPLIFl 


Oparatlonal life span of 
HSG, In months 


Same as #17 


64 




4ECL 




Confidence level for OPLIFl 




65 




43 


ACTIVE 


Still operational? 


0 - No 

1 ^ Yes 

9 No Response 


66 




43CL 




Confidence level for ACTIVE 




67 




45 


OPFRQl 


Operational ube frequency 
per year (average) 

• 


0 - Not operational 

1 ^ 1=5 

2 - 6-10 

3 - 1M5 

4 - 16-20 

5 - 

6 = 26-50 

7 - 5U100 
8-101 or rnore 
9 - Nd Rtsponsi 


68 




46 


IXPUSE 


Usfid for experimental 
purposes? 


0 - No 

1 - Yes 

9 - No Response 


69 




48 


iXPRP 


- Was the MSG intended to 
be used experirentally? 


Same as above 


70 




49 


IDUSE 


Used for iducational 
purposes f 


Same as above 


71 




51 


iOPRO 


Was the MSG Intended to 
be used educationally? 


San^ as above 


72 




52 


TRAf^SU 


Level of difficulty of 
tfinsferal of MSG 


0 - Not for transfer 

1 - General ly 

2 - Moderate 

3 - Middling 

4 = Difficult 

5 - Ixtrsme 

9 - No response 


73 



ERIC 
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Cnrd j — c: o n c i n u e d 





Variable 
Numbe r 


Variable 
Ninie 


Oiflnitlon 


Attrlbutis 


Col Uffin 


•^^SCi Co^ts to 
Ot^rate, cont. 


13 


THANK 


Trrinsfir CDits to Operate 
Confidince level for TRAfiSC 


0 " 'M^iUnknnwn 

1 - Cost 

2 ^ +10, 

3 - +10^25 

4 . +25=50 

5 - ^10^100 

6 - >ido . 


74 
75 




55 


LUP>ISG 


Is thire an flSC that Serves 
Sin-e purpQSi? 


0 - Uq 

1 ^ Vis 

9 - No Response 


76 


.^ecoinimnddtions 
ypi nioni 


57 . 


CLEAR 


Clearinghouse utility 


0 - Hiohl^ useful 

1 - Useful 

2 - Same 

3 - Harniful 

4 - Highly harmfu] 
9 - tiO ngiponii 


77 




b8 


STNFDl 


Standardi nation "5 utility 


0 - Highly useful 

1 - Useful 

2 " Same 

3 - Hamiful 

4 - Highly Harmful 
9 - No Rfiponii 


78 






HEGCEN 


Opfratfon utility of 
regional center 

[coded from verbal feiponse] 


Same as abovi 


79 




60 


EXREV 


Opinion on Ntill ty of 
^external review board 

[coded from verbal response] 


Same ai above 


80 
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Uf t Hi 



V 3 r i i3 :5 1 p 



"1 fnitif iUii'f' af *'SL 



f 1 nt^ U%i-^ to ^?i|upncd 



9 - 



iPf[':\ 1 for' NPtl 1 y*^ H 
1-1 1 ril- 1 

f 1 it-d 

in t;i: 1 y f ni'u ri'*>SJ^(' 
Tea I Ti-^ i' 



? ^ Futurr 

f , ( C"V\ Pi! t ^ fipi { 



ERIC 
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Card 4 — continued 



General 
Class 


Vaf iflbli 
Numbir 


Variable 


Definition 


Attrlbytas 


Column 


MSu Characttri zation 
and Description* 
cont . 


68 


LRTlf'E 


Level of resolution, modil 
tlmt, sn^l list uni t 


0 - 1;A, Unknown 

1 - Seconds 
R - Minutss 
3 - Hours 

^ - Days . 

5 - Weiki 

6 - Quarteri 

7 - Ytars 

8 - > Years 


13 




69 


LRSPCE 


Livil of resolutlDni imillait 
modfl spatial count of 
analysi s 


0 ^ NA; Unknown 

1 - Meters 

2 - Kllometiri 

3 - Thiatir/contintnt 


14 


- 


69CL 




Confidents Levt1 for LRSPCE 


4 - Variid 

5 - iIq rssponie 


15 


■ 


70 


LRSIDE 


Level Qf resolutiDn, sides In 
MSU 


0 ^ NA;Unknown 

1 ^ Individuals 

2 * Small groups(struct. ) 

3 - Smil 1 Brgi , 

4 - Largi orgs. 

5 - Very large orgs, 

6 - Small gfoupiCunstfuct, ] 
9 ^ iio Response 


16 




71 


LRMIL 


Level of reioiutisn, mnitary 
act ion 


0 - NAjUnknQivn 

1 ^ Engagiment 

2 - Battle 

3 ^ Carpaign 

4 " War 

5 - Diplomatic 


17 




71CL 




Confidence level for LRMIL 




18 




72 


mm 


Aft random ivints consldired? 


0 - ho 

1 - Yes 

9 - No Risponsi 


li 


MSij Plinrrirg Factors 

Af\ii fist s 


73 


PATAS 


Where did data come from? 

- 

i 


0 - NA| Unknown 

1 ^ Military/not cross- 

chicked 

2 - Mill tary/croiS-chfeckid 

3 ^ Civil 

5 - Qenfrated own 

1 - Field txercist 

? - Combination of abova 


20 








1 


^ - Ho Rtiponii 
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Card 4 — continued 



Giniral 
Clais 


Variibli 
Hum be p 


Variible 
Nime 


Oefinlttan 


Attributei 


Column 


MSG PUnnlng 
FaEtdri and 
Datii cpnt. 


78 


SENSE? 


Was iiniitiylty analysis doni 


? 0 - No 
1 - Yii 

9 - No ^ispenii 


11 




eo 


LATAV 


Who validated the data, how 
wtil [cQdid from text] 


0 - NAtUnknQwn 

1 - High quality 

2 - ttodfrate 

3 - Weak 

4 - UqI dene 

9 ^ No Relponie 


22 




B2 




Whit language is MSG 
wri t tin In ? 


0 - NA 

1 - FORTRAN 

2 - PL^l 

3 - COBOL 

4 - SPSS 

5 - SiftSCniPT 

6 - ALGOL 

7 - ASSEfttLER 
a - Othtr 

9 - No RiSponse 


13 




BZLl 




Confidence liVil for LA.NG 




24 




83 


MSGS I Z 


How many cmputtr Instruc- 
tions (lodo'i)? 


0 - iN A. Unknown 

1 - oooo 

I - 1-2K~ 

3 - E-4K 

4 . 4-aK 

5 ^ l-iK 
fi - 8-lOK 
7 - >10K 

9 - Nq Relponie 


25 




84 


FACiL 


Spiclal farillties needed 
for MSG 


fl - f;A, Unknown 

1 - Special building 

Z ^ Didicatid computtr 
(unclat. ) 

3 - Dedicatid eompytip 

(claiiified) 

4 - Special language, 

library, gr syittm 

5 - No ReipofiSi 






B4CL 




Confidtnci level for FPQll 




27 


Preductlon 
costs 


81 

ascL 

86 


UEVHVI 


Davilppment tlmi, in months 

Confidinci level for DEVTM2 

Total man ytari In divtlop- i 
mnt 

L 

1 

.. h 


Sami ai ^17 

D ' NAiUnknown 
! - 

i - 6-10 
\ - 1UI0 

^ No ReiKonie 


21 
29 
30 



;eric 



C n r d h c o n t i n u 



General 
Cldis 


Variable 
Numbf r 


Variabli 
Nanie 


Def i n1 tion 


Attributes 


Column 


MSG Production 

Costs, CQfit. 


feSCL 




Confidence ii^vil for PLVMY] 




31 




87 


DVPMYl 


• Total profesiional man-years 
in devilopment 


Sariie as ^86 


32 




mcl 




Confidence level for rA'Pi'Yl 




33 




89 


DVPRGl 


Total progranrjter man -years 
in development 


Safne as ^86 


34 


MSG 

Uocumentati on 


90 


DOCEXT 


Extent of documentation, 
general asiessnient 


0 - NA; Zero: Unknown 

1 - Excellent 

2 - VG 

3 - Average 

4 - Weak 

5 - Poor 

6 - UnevEn/variable 

7 - Unavil lable 

8"- Combinations of above 
9 - No Rispgnse 


35 




91 


DOCLOC 


Where is the documentation 
located? 


0 - NA; Unknown 

1 - Out of print/Unk, 

2 - Proprietary /NFP 

3 " Proprl etary/Connierci a 1 

4 - Proprietary/Author 

5 - Prop, (Class . )/Author 

6 - Public/Ooc 

7 - Publlc/Loc 

8 ' Public/JDurnaU, books 

9 - No RiSpOnSi 


36-38 [3] 




93 


DOCPUB 


Publi cation type of 
documentation 


0 - Booki/articles 

1 - User manual 

2 - Program deck 

3 - Reports 

4 - 0 & 1 
5-^112 

6 - Other combinations 
9 - No Response 


39 


etnni'cal 

Keconiii-endi tioris/ 
;;Dini ons 


9S 


TECH 


Techni cal coordlnati on 
opinion 


0 - Highly undesirable 

1 - Undesirablt 

2 - Indifferent 

3 - Desirable 

4 - Hi ghly desi rable 


40 










J - No ResponsR 
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Card 4--concinued 



Technical Rec^ 
omminditlpns » cont. 



Quel tlonnal ri 
Eva ludtion 



Variabli 



91 



97 



VapT able 



STNRD2 



gUEVAL 



Types of data 



Intangibles i 
Afiumptions itijde 
.or ytl 1 ized 



Strvi ci 



U§i Datt 



136 



74 



IOC 



VOLAS 



77 



IS 



INTASP 



SER^IC 



USEDAT 



Dif Inliisn 



Standirds Cari'iittee 
[coded froffi vfrbal tsm] 



Respondent eyaluation 
of qui^ ti Qnna i re 



Riiiarchiri" aliesifnent of 
the quality of thi qutition^ 
naifi rtiponses 



Did thi refpondint pravi'di 

informatiQn In the voluntary 

asseiinitnt portion of tht 
quistionnai ri? 



Typis of data 



Intinglblis: 
AiSumptloni 
Made in Model 



Strvice Using tht MSG 



USE Initiation diti 



0 = E^cel lent 

1 - Good 

2 - Modal 

3 - Poor 

4 - iad 

9 - No response 



Same as above 



0 - No 

1 - ViS 



NA'tUnknown 



1 - Type 1 
I - Type 2 



inly 
nly 



Type 3 only 
Typil 1 and 2 
TypfS 1 ind 3 
Types 2 and 3 
All Typis 
No response 



Ho 

yts 

Could not 
dttirffiine 



9 * No risponsi 



USA 

USAF 

USN 

Qthif DOD 



0 ' Nfver opirittd 

1 - *57 

2 - 58-59 

3 * 60-61 

4 - fi2-S3 

5 - 64-65 

6 - 66-67 

7 ' 68-70 

8 ' 71-72 

i - NA, Unknown, 
No responsa 
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Card 5- 



fiintral 
Class 


Variablt 


Variable 
Name 


Difini tion 


Attributes 


Col umn 


Office 




Card 




Integer 


1-2 


Office Usi 




ID 


Siquirtce number of the 
observati on 


I nteger 


3-5 


Production 
Coits 


m 


CSTDRS 


Di rict costs to bul Id 
(lOOO's) = 


Real 


6-10 




35AS 


CSTFMi 


01 rect costs to bui Id 
fami ly of models 
(lOOO'i) 


Real 


11-15 




3H 


FUNDRS 


Direct funding to build 
(lOOO'i) 


Real 


16=20 




38S 


TOTS 


Total of al 1 types gf 
costs to build fiSG 
(lOOO'lj 


Real 


2U25 


Operational 
Losts 


39S 


. CSTOPS 


Averagi annual operating 
costs (1000*i) 


Real 


26-30 




40 S 


CSTRNS 


Cost per llngli run 
( 1000* s) 


Rial 


3U3S 




41$ 


CSTUPS 


Annual update coits 
(lOOO's) 


Real 


"SC Ah 




42R 


OPLIFI 


Number of manths 
HSG Qpirational 


Integer 


41-42 




45K 


0PKRg2 


Number of times (total) 
MSG has been run for any 
5 years 


I nteger 




Uata 

kequi renjentl/ 
Size 


75A 


INCON 


Numbir of input constants 


inttgir 


46-4i 




75y 


IHPAR 


Number of Input paramiters 


Integer 


49-11 




75C 


INVAR 


Nurrtaer of input variables 


Integer 


52-54 




76 


OUTVAR 


Nymfair of output variables 


Integer 


5S-57 


ueve iopment 
Tifoe 


B5R 


LEVTM3 


Number of months MSG under 
development 


Integer 


58-59 




86k 




Number of total man-yean 
under development 


Integer 


60-63 




B7R 


DVPMV2 


Number of professional 
man-ytari undgr divelop^ 
ment 


Integer 


64-67 




89 R 


UVPRGi 


Nun^er of prografuner 
man«years undtr develop- 
ment 

^ — . I_ 


Integer 


68-71 



ERIC 
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Cards 6-12 (narrati d 



Card 6: 



Card 7: 

uf f uu 
Uff ice 



bp i fluff's 



C^ard 8: 

uf f i 
Off ite 



{'Cfit fits ' 



Card 9; 

Iff Kf 



31 



i4 



Nine 



ID 

SPIN 



Card 
IP 



Card 



RNFIT 



01 t rCit )0t] 



i>l %i f i it jc)n 



Via- UfMofi^d^* 



Irpyrijnwu r/f the MSG tp 



Criturici liiti'd ai 
benuf i ts for hdvtnrj used 
this particuldr MSG 



rial iQri th 



I n tenor 



41- 



ERIC 



-160- 



Cards 6-12 — continued 



Gentril 
Class 


Variablt 
Numbtr 


Variable 
Name 


Definition ^ 


Attributes 


Col urnn 


Card 10: 
Office 




Card 




Intiger 




Off lei 




ID 


Sequenci numbir Of the 
obiirvatlsn 


In tiger 


3^5 




44 


USERS 


Who usei thi nSG at this 
time? 


Hollerith 


6-40 




54 


OBSOL 


Cstlmatid speed at which 
respondent thinks the model 
will beconie obioleti; reisons 
Tor this , If noted 


Horierith 


41-80 


Card 11: 












Office 




Card 




Intiger 




Uf f i ce 






Siquinci numbgr of the 
observation 


Integer 


3-5 


Uata 


73A 


DAT VI 


Soufdis of thi data used 
in thi MSG 


Hollirlth 


6^40 




80A 


0ATV2 


Procidures uiid to validate 
thi data 


Hoi lerf th 


41-80 


Card 12: 












(jffici 




Card 




Integer * 


U2 


uff ici 




ID 


Sequence number of the 
observation 


Intiger 


3-5 


Languages/ 
Machinery 


Bl 


LANGS 


Languages used to code 
thi^s MSG 


Hollerith 


6^40 




82A 


COMPS 


Computirs on which this 
MSG is (hai bien) run 


Hollerith 


41-80 



